Land backend pack-11 continuation: Go miner with exact parity, banknote 12-point integration, DC checkers with export --pairs, editor arm configs, and additive wave snapshot; clear the rubezh-2 debt with a three-item fix list
This commit is contained in:
parent
88b5e7b47d
commit
13c4375e2a
40 changed files with 3731 additions and 202 deletions
250
backend/PACK11_CONT_REPORT.md
Normal file
250
backend/PACK11_CONT_REPORT.md
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# PACK11_CONT_REPORT — бэкенд-сессия ПРОДОЛЖЕНИЯ, резидуал R1–R5 (2026-07-19)
|
||||
|
||||
> Продолжение `PACK11_REPORT.md` (Block A, `85f5b9e`). Строил по `docs/architecture/11-implementation-plan.md`
|
||||
> (ратифицирован D39.12) + `PACK11_REPORT.md` §Резидуал (пошаговые дизайны R1–R5). **НЕ коммитил** — сдача
|
||||
> на приёмку. Верификация исполнением: `go build ./... && go vet ./... && go test -race ./...` — ВСЁ
|
||||
> ЗЕЛЁНОЕ на каждом чекпойнте (логи в §Верификация). Block A НЕ трогал (под тестами).
|
||||
|
||||
## 0. Санкционированный порядок и итог
|
||||
|
||||
**Развилка порядка эскалирована владельцу/оркестратору в начале сессии** (R1 = крупный каскад, меняющий
|
||||
ГРАНУЛЯРНОСТЬ вывода → ~весь e2e-суит + read-models + golden; риск красного дерева vs жёсткое правило
|
||||
«суит зелёный на каждом чекпойнте»). **Оркестратор санкционировал вариант 1:** R2→R3→R4→R5 сначала (все
|
||||
самодостаточны и зелёно-достижимы независимо от R1), **R1 — капстоуном на остатке бюджета**, с условиями:
|
||||
чекпойнт-ревью после R2+R3 и после R4; R3 на текущем последовательном раннере; каждый снапшот-двигающий
|
||||
лендинг — свой golden re-capture с masked-диффом, СТРУКТУРНЫЙ рерайт — один раз в R1; R1 сначала аддитивные
|
||||
под-шаги без каскада, переключение драйвера — финальный атомарный шаг; не дотягивает до зелёного → дерево
|
||||
зелёное, R1 точным резидуалом (запланированный исход).
|
||||
|
||||
| WS / шаг | Статус | Верификация |
|
||||
|---|---|---|
|
||||
| **R2 · WS3 Go-майнер** | ✅ **СДАНО** | Go↔Python паритет **EXACT**: 13618 канд-СЕТ, катастроф {方源:0,蛊:1,蛊师:2,古月:22}, recall@proposed **0.9649**; alias-ярус-1 (5 R4-блоков+компози-гард); сид-дельта+`MinedDeltaYAML`; `Source:mined`-путь; `tmctl seed-lint`. |
|
||||
| **R3 · WS4 банкнота 12 точек** | ✅ **СДАНО** (на посл. раннере) | срез-до-classify (fresh+resume); `tm-banknote-v1:` derived + re-point после эскалации; `banknoteSnap` (omitempty); v9-телеметрия; finish=stop-only гейт. 5 e2e-пинов. |
|
||||
| **R4a · `tmctl export --pairs`** | ✅ **СДАНО** | source-столбец (пред-условие §5(д)-FP DC1/DC2). |
|
||||
| **R4b · чекеры DC1/DC2/DC6 + zh-ru пакет** | ✅ **СДАНО** | эмпир-позитивы пойманы; DC2 **байт-верно референсу** (ok_re-supress + case-sensitive — фикс ревью); `cheapGateVersion` v3; golden re-capture masked-чист. |
|
||||
| **R4c · DC3 gender-инъекция** | ✅ **СДАНО** | `memoryEntry.gender`→editor-констрейнт; `renderFormatVersion` v2; golden re-capture masked-чист. |
|
||||
| **R5 · WS6 арм-конфиги** | ✅ **СДАНО** | `pipeline-arm-mistral.yaml`/`pipeline-arm-deepseek-pro.yaml`; `echo_mine_test` под варианты; rate-guard связка. |
|
||||
| **R1 · волновой ИСПОЛНИТЕЛЬ** | ⚠️ **АДДИТИВ СДАН · драйвер-свитч РЕЗИДУАЛ** | `snapshotIDForWave` (пер-волновой снапшот, «переоплата ОДНА» пиннута) + W0 sticky-precompute (**байт-идентично, golden-доказано**) — СДАНЫ, зелёные. Параллельный dispatch W1/W1.5/W2 — резидуал (§R1-residual, дизайн ниже). |
|
||||
| **Чекпойнт-ревью** | ✅ ×3 | R2+R3 (4 minor/nit, фикс) · R4 (2 DC2-parity, фикс) · финал (см. §Ревью). Адверсариал-воркфлоу, author≠reviewer, refute-by-default. |
|
||||
|
||||
---
|
||||
|
||||
## R2 — WS3 Go-майнер (план §3) — ✅ СДАНО
|
||||
|
||||
Новый пакет `internal/pipeline/miner*.go` (substrate/detect/patterns/palladius/alias/emit) — **байт-верный
|
||||
порт frozen exp16** (`exp16_common/detectors/patterns/arms/alias/palladius/emit_owner_sheets.py`),
|
||||
**дефолт-лемматизатор B**. Per-буллет (§R2):
|
||||
|
||||
- ✅ **Порт V-A→V-C по ТОЧНЫМ порогам §3(б):** c-value `g(L)=log₂(L+1)` (не log₂L — пин
|
||||
`TestMinerLengthMultNonDegenerate`/`CValueSingleCharNested`); subsumption α=0.80 (`TestMinerSubsumption`);
|
||||
weirdness `p_book/max(word_rel,char_indep_rel,1e-9)`, N_book = полная пре-флор масса; n-граммы Han 1–6
|
||||
overlapping, freq-floor 3; паттерн-каналы (surname/百家姓−凝, title-suffix/bare, ordinal_title, rank_grade,
|
||||
topo, авто-формант over_rep≥15∧partners≥3 — `TestMinerFormantDetection`); arm A3 λ=0, bonus·pw,
|
||||
subfloor `bonus+40·f·pw`. Детерминизм: все сорты `(-score,-freq,src)`, map-итерация только ПЕРЕД тотал-сортом.
|
||||
- ✅ **Дефолт B — дроп Палладий-подканала V-C** (плюс `dst_variants`, `is_name_lemma`, spread λ, alias-R3,
|
||||
canon-dst-recovery — все pymorphy3-зависимые; dst доставляет банкнота WS4). `is_palladius_token` порт как
|
||||
**детерминированный** артефакт (Палладий-таблица, без pymorphy3) + тестируем (`TestMinerPalladiusToken`).
|
||||
**Палладий-ЗАВИСИМЫЙ ранг 古月:21→22** — как предписано §10-14/§3(д); катастроф-скрин ∈top-50 PASS.
|
||||
- ✅ **Палладий-ИНВАРИАНТНЫЕ гарантии — EXACT** (`TestMinerFullBookParity`, data-gated `TM_MINER_PARITY=1` на
|
||||
стенд-данных: jieba-контраст + `rerun/records.json` + `guzhenren-seed-v2.yaml`): **n=13618** (SET-членство),
|
||||
recall@proposed **0.9649** (55/57 GT), катастроф {方源:0,蛊:1,蛊师:2,古月:22}. Скипается при отсутствии данных
|
||||
(как `TM_LIVE`). **Python = референс: byte-verified, расхождений 0 по инвариантам.**
|
||||
- ✅ **Алиас-ярус-1 (R1–R4, R4 hard-blocks ПЕРВЫМИ):** `TestMinerAliasRules` (5 сабтестов): R1-extension
|
||||
(方源⊂古月方源), компози-гард (古月+族长=фраза-не-алиас, но 古月+方源=fullname-алиас), R4-iii/v (族长≠四代族长 по
|
||||
разному approved-dst), R4-ii (разный пол блок, hidden не блокирует), R4-i/R2 (одна фамилия+разные имена→family).
|
||||
**R3 (shared-ru-rendering) ОМИТ в дефолте B** (нужен `dst_lemmas`/pymorphy3) — задокументировано в коде.
|
||||
- ✅ **Эмиссия сид-дельты §C2-7:** `MineBank`→`MinedTerm` с фильтрами `build_precision30` (тип∈{name,place,title},
|
||||
freq≥5, ∉subsumed, len≥2, ∉фрагмент по PARTICLE) — НЕ сырой 13618-дамп; alias-кластеризация (репрезентант =
|
||||
топ-ранг; кластер с seed-поверхностью = алиас-of-existing, не эмитится). `MinedDeltaYAML` → seedTerm-схема
|
||||
БЕЗ новых полей. `TestMineBankDeterministicAndNonSeed`/`TestMinerEmissionFilters`.
|
||||
- ✅ **`Source:"mined"` mined-write-путь:** `minedToCandidates` (штамп `Source:mined`, status:auto, no-dst,
|
||||
зеркало `rubyToCandidates`, НЕ через `loadGlossarySeed`). `TestMinedToCandidatesStampsMinedSource` +
|
||||
`TestMinedDeltaStaysBaseBankStable` (Source:mined НЕ двигает `BaseVersion` — «переоплата ОДНА» через реальный путь).
|
||||
- ✅ **`tmctl seed-lint`:** `SeedLint(path)` = сухой прогон РЕАЛЬНОГО `loadGlossarySeed`+`approvedSharedKeyCollisions`;
|
||||
CLI `seed-lint --seed <yaml>` (без `--config`). `seedlint_test.go`: чистый→OK; дефекты (approved-без-dst /
|
||||
unknown-status / duplicate / shared-key) → фейл-лауд; **эмитированная дельта линтует чисто** (`TestSeedLintEmittedMinedDelta`).
|
||||
|
||||
**Артефакт-зависимость:** прод-майнер грузит jieba-контраст (5MB, SHA `7197c321…`, ВНЕ git) через
|
||||
`LoadContrast(io.Reader)` — путь конфигурируется вызывающим. Тесты — синтетический малый контраст;
|
||||
паритет — реальный jieba со стенда.
|
||||
|
||||
**Не-каскадная зона R2:** `MineBank`/`minedToCandidates`/`MinedDeltaYAML` — библиотека пак-3; ВЫЗОВ на
|
||||
W1.5-границе (реseed через `ReplaceGlossary`) — часть волнового W1.5-стопа (R1-residual). Сдано как
|
||||
самодостаточное, тестируемое ядро (санкция оркестратора: R2 самодостаточен, R1 вайрит).
|
||||
|
||||
---
|
||||
|
||||
## R3 — WS4 банкнота: 12 интеграционных точек (план §4) — ✅ СДАНО (на посл. раннере)
|
||||
|
||||
Ядро `banknote.go` (Block A) + интеграция. **Все точки пер-стадийные → в волновую модель переезжают КАК ЕСТЬ**
|
||||
(применяются к W1/draft-стадии; условие оркестратора). Per-точка:
|
||||
|
||||
1. ✅ **Санитайзер:** банкнота-срез — ПЕРВЫЙ рубеж (`applyBanknote` до `classifyOutput`); trailing-note НЕ
|
||||
реюзим (ревью-1 F3). Финал-бэкстоп = телеметрия п.10.
|
||||
2. ✅ **cjk_leak:** src-колонка Han по построению → срез ДО санитайзера снимает ложь.
|
||||
3. ✅ **classify/echo (п.3, ЖЁСТКИЙ порядок):** `applyBanknote` режет блок на ОБОИХ путях runAttempt (fresh-call
|
||||
И checkpoint-hit/resume) ДО `classifyOutput` → echo/coverage/length видят ОЧИЩЕННЫЙ перевод, не Han-сноску.
|
||||
Пин `TestBanknoteClassifyOverCleaned` (клин RU + Han-тяжёлый блок → OK, не cjk_artifact).
|
||||
4. ✅ **Coverage:** числитель по очищенному (срез раньше coverageCheck).
|
||||
5. ✅ **max_tokens += `bankTokenBudget`** (≤12×12) для translator при enabled (перевод первым, блок хвостом →
|
||||
трункация бьёт блок). Пин: draft `max_tokens=656` (512+144) в e2e.
|
||||
6. ✅ **request_hash/снапшот:** `banknoteSnap{enabled,parser_version}` в `snapshotID()`. **ДЕВИАЦИЯ-с-обоснованием:
|
||||
omitempty-указатель (nil при off) вместо безусловного `{enabled:false}` зеркала sanitizerSnap** — включение
|
||||
канала И bump парсера ВСЁ РАВНО громкий сдвиг (несущая loudness §4а-п6 сохранена), но ДОБАВЛЕНИЕ фичи НЕ
|
||||
инвалидирует banknote-OFF книги (в т.ч. golden-фикстуру) → R3 golden БАЙТ-СТАБИЛЕН, re-capture не нужен.
|
||||
Пин `TestBanknoteSnapshotFold`.
|
||||
7. ✅ **Чекпоинт/резюм:** derived content-addressed → resume re-derives id бесплатно; сырой чекпоинт хранит
|
||||
RAW-ответ, resume пере-режет.
|
||||
8. ✅ **Re-point (главная правка):** OK-путь draft `final_hash` → `tm-banknote-v1:` derived-чекпоинт с ОЧИЩЕННЫМ
|
||||
черновиком (`commitBanknoteExport`, формула §4б `sha256("tm-banknote-v1\x00"+reqHash+"\x00"+stripped)`); стоит
|
||||
ПОСЛЕ резолва эскалации, потребляет `last.bankStripped` (эскалированный черновик тоже очищен). Пины
|
||||
`TestBanknoteSliceReTelemetryResume` (final_hash=derived, derived=clean, resume $0) + `TestBanknoteEscalatedDraftStripped`.
|
||||
9. ✅ **Пост-чек глоссария:** не затронут.
|
||||
10. ✅ **Телеметрия (лауд):** store v9 (`n_banknote_lines`/`banknote_parse_fail`/`banknote_truncated`) в
|
||||
retrieval_state; тред draft→translateChunk→persistRetrievalState.
|
||||
11. ✅ **DeepSeek thinking-ON нетронут** (echoMineViolation).
|
||||
12. ✅ **Экспорт:** final без сноски автоматически (срез раньше).
|
||||
|
||||
**§4б:** толерантный парсер (Block A) + **finish=stop-only гейт** — кандидаты ПАРСЯТСЯ только при `finish=="stop"`;
|
||||
срез идёт РЕГАРДЛЕСС (echo/length classify всегда видят clean). Пин `TestBanknoteFinishStopOnlyGate`.
|
||||
|
||||
---
|
||||
|
||||
## R4 — WS5 чекеры + данные пакета + `export --pairs` (план §5) — ✅ ЯДРО СДАНО
|
||||
|
||||
- ✅ **R4a `tmctl export --pairs`:** `Export(pairs bool)` + `ChunkExport.Source` (omitempty, $0 из манифеста);
|
||||
дефолт-экспорт target-only (обратно-совместим). CLI `--pairs`. `TestExportPairsIncludesSource`.
|
||||
Пред-условие §5(д)-FP DC1/DC2 (src↔target). **Read-only проекция — снапшот/wire/деньги не трогает.**
|
||||
- ✅ **R4b DC1/DC2/DC6 + zh-ru пакет** (`checkers_zh_ru.go`, порт `ws5_checkers_verify.py`): наблюдаемость
|
||||
(НЕ диспозиция), в `runCheapGates`→retrieval_state; самогейтятся на контенте (0 на не-zh/чистом). **DC2 —
|
||||
байт-верно референсу** (после ревью-фикса): ok_re-suppression гард (数十万 не фаерит при верном «сотни тысяч»)
|
||||
+ case-SENSITIVE inner (референс без re.I). `TestDC1/DC2/DC6` (эмпир-позитивы + non-firing + parity-кейсы).
|
||||
- ✅ **R4c DC3 gender-инъекция** (§5(б), «фикс Бай Нинбин»): `memoryEntry.gender`←`row.Gender`;
|
||||
`genderConstraintNote` в `renderEditorConstraintBlock` (male/female жёсткие формы; hidden — мандат безродовых);
|
||||
ДАННЫЕ пола уже в `MemoryVersion`, РЕНДЕР — в `renderFormatVersion` v2. `TestDC3GenderInjection`.
|
||||
- **Пакет данных = код-консты, версия `cheapGateVersion`** (та же дисциплина, что существующие cheap-данные):
|
||||
правка данных = громкий resnapshot. Конфиг-загружаемый per-pair пакет — чище, **резидуал** (§Резидуал).
|
||||
|
||||
**Golden re-capture ×2 (masked-чист, §Golden):** (1) `cheapGateVersion` v2→v3 — единственная payload-правка =
|
||||
`style_check_version`, wire/вердикты байт-идентичны после hash-маскинга; (2) `renderFormatVersion` v1→v2 —
|
||||
payload-версия + 7 editor-wire-тел получают gender-note (gender:m терм фикстуры), НИ ОДНОЙ вердикт/final-правки.
|
||||
|
||||
---
|
||||
|
||||
## R5 — WS6 арм-конфиги (план §6) — ✅ СДАНО
|
||||
|
||||
- ✅ **Армы = КОНФИГ:** `configs/pipeline-arm-mistral.yaml` (editor=mistral-large-2512) /
|
||||
`pipeline-arm-deepseek-pro.yaml` (editor=deepseek-v4-pro, `few_shot:false`). Оба — копии боевого C1 с
|
||||
ЕДИНСТВЕННОЙ правкой editor-model → другой `stageSnap.Model` → другой снапшот (арм-механика Block A). draft/
|
||||
translator неизменны.
|
||||
- ✅ **`echo_mine_test` под варианты** (`TestSwapArmConfigs`): каждый арм грузится; editor=арм-модель; НИ ОДИН не
|
||||
вооружает эхо-мину (deepseek-v4-pro → ReasoningNone, thinking ON, `echoMineViolation` пуст на echo-prone
|
||||
провайдере); few_shot (deepseek off / mistral nil=ON); **mistral-large-2512 несёт `rate_limit`** (WS6 пред-условие —
|
||||
свап-арм за rate-guard WS1b).
|
||||
- ✅ **Атрибуция арма:** editor-model уже в `chunk_status`/checkpoint `model_actual` → export/report несут её
|
||||
БЕЗ новой кодовой правки (план: «уже в chunk_status»). Пере-прогон армов = отдельные джобы (разные снапшоты).
|
||||
|
||||
---
|
||||
|
||||
## R1 — волновой ИСПОЛНИТЕЛЬ: аддитив СДАН, драйвер-свитч РЕЗИДУАЛ
|
||||
|
||||
**Санкция оркестратора:** сначала аддитивные под-шаги без каскада; драйвер-свитч — финальный атомарный шаг;
|
||||
не дотягивает → зелёное дерево + точный резидуал (**запланированный исход**). Сделано:
|
||||
|
||||
- ✅ **Пер-волновой снапшот (`snapshotIDForWave`, §1в — несущее):** рефактор `snapshotID()`→`buildSnapshotID(stages,
|
||||
memVersion)` (**байт-идентичен** для all-stages+enriched → golden/последовательный драйвер НЕ двигаются).
|
||||
`snapshotIDForWave(W1)` = translator-стадии + BASE-версия (excl mined); `(W2)` = editor-стадии + ENRICHED.
|
||||
`waveStages` (партиция по роли), `baseMemoryVersion`. Пин `TestSnapshotIDForWave`: W1≠W2; **mined-approved
|
||||
добавка двигает ТОЛЬКО W2, W1 стабилен** («переоплата ОДНА» на снапшот-уровне) + `TestWaveStagesPartition`.
|
||||
- ✅ **W0 sticky-precompute (`wave.go precomputeSticky`, §1б):** sticky-цепочка — кросс-чанковая зависимость
|
||||
(не пере-считать внутри параллельной волны) → предвычислена в W0 одним проходом (пер-глава reset, `Select`,
|
||||
advance). **Последовательный драйвер ТЕПЕРЬ ЕЁ КОНСУМИТ** (`bookrun.go` W0-precompute; `translateChunk`
|
||||
принимает готовый `memSel`, снят inline-`Select` + `activeIDs`-return). **БАЙТ-ИДЕНТИЧНО** — `TestGolden`
|
||||
зелёный без правок (несущий байт-паритет-пруф). Это ТОЧНЫЙ precompute, который консумит волновой исполнитель.
|
||||
|
||||
**РЕЗИДУАЛ R1 — драйвер-свитч (точный дизайн, `PACK11_REPORT.md §R1` + план §1):**
|
||||
1. Партиция стадий по волне — **ГОТОВА** (`waveStages`); пер-волновой снапшот-апсерт (два UpsertSnapshot) —
|
||||
рефактор `bookrun.go` W0-прелюдии (сейчас один `snapshotID`; заменить на `snapshotIDForWave(W1/W2)`, пинить
|
||||
стадии волны к своему через `EnsureJob(snapshot_Wx)`).
|
||||
2. W1-dispatch: N воркеров тянут draft-чанки, `runStage(draft)` под `snapshot_W1`, `memSel`=precompute[i] —
|
||||
**ГОТОВЫ**: precompute, rate-guard (Block A), single-writer деньги (Block A), eager-clients (Block A).
|
||||
3. W1.5-стоп: пустая mined-дельта → авто-продолжение; иначе `MineBank`→карта-подписи→подпись→`minedToCandidates`
|
||||
→`ReplaceGlossary`(reseed)→`snapshot_W2` — **компоненты ГОТОВЫ** (R2), нужна оркестровка стопа/резюма.
|
||||
4. W2-dispatch: N воркеров тянут edit-ЕДИНИЦЫ (`EditUnitID`, Block A), `prev`=конкатенация draft-чекпоинтов
|
||||
единицы по `final_hash`, СВЕЖИЙ `Select` по тексту ВСЕЙ единицы над enriched-банком, `runStage(edit)` под
|
||||
`snapshot_W2`. **W2 НИКОГДА не пере-рендерит draft** (несущий инвариант — редактор читает по `final_hash`).
|
||||
5. **Каскад (почему не в этой сессии):** edit=единица меняет ГРАНУЛЯРНОСТЬ вывода (edit per-UNIT, адресуется
|
||||
`(chapter, firstChunkIdx, "edit")`) → `ChunkOutcome`-форма + export/status/quality read-models + ~весь
|
||||
e2e-суит (asserts per-chunk 2-стадии + call-counts) + СТРУКТУРНЫЙ golden-рерайт. Оркестратор явно
|
||||
санкционировал структурный рерайт «один раз в R1»; аддитив-пруф (golden байт-стабилен) даёт уверенность,
|
||||
что все несущие R2–R5 компоненты на месте для атомарного свитча в отдельной сессии/окне.
|
||||
6. kill-9 с N резервами: порт `store/kill9_test.go` на N воркеров (money-инвариант Block A уже single-writer-safe).
|
||||
|
||||
---
|
||||
|
||||
## Открытые эскалации / решения (флаг оркестратору при лендинге)
|
||||
|
||||
1. **Дефолт-B майнер** — ратифицированный §10-14 дроп pymorphy3-подканалов (spread λ / dst_variants /
|
||||
is_name_lemma / Палладий-конфирм / alias-R3 / canon-dst). 古月:22 (не 21). Все Палладий-ИНВАРИАНТНЫЕ гарантии
|
||||
воспроизведены EXACT. dst доставляет банкнота. **Faithful-A (полный pymorphy3-порт) — только при готовой
|
||||
поддерживаемой Go-морфологии** (не строил).
|
||||
2. **`banknoteSnap` omitempty** (не безусловный sanitizerSnap-зеркало) — девиация-с-обоснованием (§R3-п6):
|
||||
loudness сохранена, off-книги не re-bill-ятся, R3 golden байт-стабилен. **Требует подтверждения при лендинге.**
|
||||
3. **DC-пакет данных = код-консты** (версия cheapGateVersion), не конфиг-загружаемый per-pair файл — конфиг-форма
|
||||
резидуал (§Резидуал). Дисциплина resnapshot сохранена.
|
||||
4. **R4 резидуал (честный, бюджет→R1):** DC7 grade/role-чекер (нужен promote 甲乙丙丁/四代族长 approved — owner-only,
|
||||
§5-A2 UNVERIFIED); DC3 male/female-ЧЕКЕР (нужна coref-lite, §5(д)-FP-гейт); omission-бэкстоп (лёгкий Go-ru-стеммер
|
||||
без pymorphy3 + лемма-матч по {dst,decl.forms,aliases}); DC2 word↔word дроби (q4a rule_b1/b2). Дизайны — план §5(б).
|
||||
5. **R1 драйвер-свитч резидуал** (запланированный исход, санкция оркестратора) — дизайн §R1 выше.
|
||||
6. **§5(д)-FP-замер DC1/2/3/6** — платный пере-прогон (не в этой сессии); `export --pairs` — пред-условие СДАНО.
|
||||
⚠ FP-замер прогонять через **Go-код** (не `ws5_checkers_verify.py`), т.к. чекеры теперь байт-верны референсу,
|
||||
но лендинг-решение должно отражать что реально шипается.
|
||||
|
||||
---
|
||||
|
||||
## Структурные переносы
|
||||
- Новые файлы `internal/pipeline/miner_*.go` (9 шт — новый детектор-пакет, не рефактор), `checkers_zh_ru.go`,
|
||||
`wave.go` (W0-precompute). Все — новые оси.
|
||||
- `snapshotID()` → тонкая обёртка над извлечённым `buildSnapshotID(stages, memVersion)` (байт-идентично).
|
||||
- `translateChunk` сигнатура: `stickyPrev map[...]` → `memSel memorySelection` (готовый из W0); снят
|
||||
`activeIDs`-return (precompute владеет sticky-advance). Внутренний вызов, вне-скоуп поведение не затронуто.
|
||||
- `Export()` → `Export(pairs bool)` (6 тест-вызовов обновлены на `Export(false)`).
|
||||
|
||||
## Багфиксы вне плана
|
||||
- Нет вне-плановых. Правки по ревью: **R2+R3 (4):** derived-namespace `banknote-v1`→`tm-banknote-v1` (§4б EXACT-формула,
|
||||
отдельная const от parser_version), мёртвая `rank`-карта в `MineBank`, doc-коммент, error-wrap
|
||||
`commitBanknoteExport`. **R4 (2):** DC2 ok_re-suppression-гард + case-sensitive inner (байт-паритет референсу).
|
||||
Все верифицированы адверсариально, пофикшены, суит зелёный.
|
||||
|
||||
## Golden re-capture (masked-структурный дифф — обязан быть объясним)
|
||||
- **R4b (cheapGateVersion v2→v3):** masked-дифф = 4 строки, ЕДИНСТВЕННАя payload-правка `style_check_version`;
|
||||
wire-тела + вердикт-строки БАЙТ-ИДЕНТИЧНЫ после hash-маскинга (verified скриптом).
|
||||
- **R4c (renderFormatVersion v1→v2):** masked-дифф = 18 строк: 4 версия-payload + 7×(old/new) editor-wire-тел с
|
||||
gender-note (gender:m терм). НИ ОДНОЙ вердикт/disposition/final-text правки (Counter: version/gender-note/wire только).
|
||||
- **R3/R5/R1-аддитив:** golden НЕ двигается (banknote omitempty off; армы — отдельные файлы; snapshot-рефактор
|
||||
байт-идентичен). СТРУКТУРНЫЙ golden-рерайт — событие драйвер-свитча (R1-residual), не этой сессии.
|
||||
|
||||
## Верификация (исполнением — логи, не слова)
|
||||
```
|
||||
$ go build ./... → OK
|
||||
$ go vet ./... → OK (чисто)
|
||||
$ go test -race ./... → ВСЁ ЗЕЛЁНОЕ:
|
||||
ok cmd/tmctl · config · ledger · llm · pipeline (~28s -race) · store
|
||||
$ gofmt -l <мои файлы> → пусто (репо несёт ПРЕ-существующий gofmt-долг в не-моих файлах — не трогал)
|
||||
$ TM_MINER_PARITY=1 go test -run TestMinerFullBookParity
|
||||
→ n=13618 · катастроф {方源:0,蛊:1,蛊师:2,古月:22} · recall@proposed=0.9649 (55/57 GT) — PASS
|
||||
```
|
||||
**Фикстуры плана исполнены:** miner-паритет (Палладий-инвариант EXACT, 古月:22 дефолт-B) · банкнота 12 точек
|
||||
(5 e2e-пинов) · seed-lint 0 фейл-лаудов · DC-позитивы пойманы · пер-волновой снапшот «переоплата ОДНА» ·
|
||||
W0 sticky байт-паритет (golden) · draft request_hash стабилен (golden).
|
||||
|
||||
## Чекпойнт-ревью (адверсариал-воркфлоу, author≠reviewer, refute-by-default, 3 линзы)
|
||||
- **R2+R3:** 5 сырых → 4 confirmed (все minor/nit, 1 refuted): `tm-banknote-v1` namespace / мёртвая rank /
|
||||
doc / error-wrap. **НИ ОДНОГО major correctness/wire/money.** Все пофикшены.
|
||||
- **R4:** 2 confirmed (оба DC2 reference-divergence): ok_re-supress-гард (MAJOR — реальный FP-дивердж на
|
||||
ch2/chunk0) + case-sensitivity (MINOR). Оба пофикшены — DC2 байт-верно референсу.
|
||||
- **Финал (R5 + R1-аддитив + кросс-каттинг инвариантов, 2-мерный воркфлоу + verify):** **0 находок — ЧИСТЫЙ
|
||||
вердикт.** Обе линзы (R1-аддитив байт-паритет/пер-волновой снапшот/precompute-эквивалентность; R5-армы +
|
||||
кросс-каттинг: деньги/снапшот-дисциплина/детерминизм/store-v9) вернули пустой список. Ни одного invariant-
|
||||
break, un-folded wire/verdict-change, byte-parity-break или map-order-в-выводе. `go test -race ./...` зелёный,
|
||||
golden байт-стабилен, arm/wave-тесты проходят — clean bill.
|
||||
|
|
@ -17,9 +17,11 @@ import (
|
|||
type invocation struct {
|
||||
cmd string
|
||||
cfgPath string
|
||||
seedPath string // seed-lint: the glossary seed YAML to validate (no --config)
|
||||
resnapshot bool
|
||||
asJSON bool
|
||||
asPlaintext bool
|
||||
asPairs bool // export: include the source column (--pairs) for the DC1/DC2 FP-measure
|
||||
sel pipeline.RedriveSelector
|
||||
}
|
||||
|
||||
|
|
@ -48,18 +50,27 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
|
|||
resnapshot := fs.Bool("resnapshot", false, "re-pin existing jobs to the current config snapshot (пере-перевод оплаченных чанков — явное согласие)")
|
||||
asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE")
|
||||
asPlaintext := fs.Bool("plaintext", false, "export: emit the concatenated human text instead of the default stable JSON")
|
||||
asPairs := fs.Bool("pairs", false, "export: include the source text per chunk (src↔target column for the DC1/DC2 FP-measure, WS5)")
|
||||
chapter := fs.Int("chapter", -1, "redrive: restrict to this chapter (default: any)")
|
||||
chunk := fs.Int("chunk", -1, "redrive: restrict to this chunk index within the chapter (default: any)")
|
||||
reason := fs.String("reason", "", "redrive: restrict to this flag_reason (default: any)")
|
||||
dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything")
|
||||
seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)")
|
||||
if err := fs.Parse(rest); err != nil {
|
||||
return invocation{}, err
|
||||
}
|
||||
// seed-lint validates a standalone seed YAML — it takes --seed, not --config (no book/store/keys).
|
||||
if cmd == "seed-lint" {
|
||||
if *seed == "" {
|
||||
return invocation{}, fmt.Errorf("--seed <glossary.yaml> is required for seed-lint")
|
||||
}
|
||||
return invocation{cmd: cmd, seedPath: *seed}, nil
|
||||
}
|
||||
if *cfgPath == "" {
|
||||
return invocation{}, fmt.Errorf("--config book.yaml is required")
|
||||
}
|
||||
return invocation{
|
||||
cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, asJSON: *asJSON, asPlaintext: *asPlaintext,
|
||||
cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs,
|
||||
sel: pipeline.RedriveSelector{
|
||||
Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun,
|
||||
},
|
||||
|
|
|
|||
|
|
@ -69,11 +69,13 @@ func run() error {
|
|||
case "status":
|
||||
return status(ctx, inv.cfgPath, inv.asJSON)
|
||||
case "export":
|
||||
return export(inv.cfgPath, inv.asPlaintext)
|
||||
return export(inv.cfgPath, inv.asPlaintext, inv.asPairs)
|
||||
case "redrive":
|
||||
return redrive(ctx, inv.cfgPath, inv.resnapshot, inv.sel)
|
||||
case "seed-lint":
|
||||
return seedLint(inv.seedPath)
|
||||
default:
|
||||
return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive)", inv.cmd)
|
||||
return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|seed-lint)", inv.cmd)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -94,6 +96,17 @@ func translate(ctx context.Context, cfgPath string, resnapshot bool) error {
|
|||
})
|
||||
}
|
||||
|
||||
// seedLint validates a glossary seed YAML (a manual seed or the emitted mined delta) through the REAL
|
||||
// loadGlossarySeed fail-louds + the shared-key collision check (WS3). $0, no store, no provider keys — a
|
||||
// pre-flight the operator runs on the mined delta before the W1.5 reseed. Prints "OK" on a clean seed.
|
||||
func seedLint(seedPath string) error {
|
||||
if err := pipeline.SeedLint(seedPath); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(os.Stdout, "seed-lint OK: %s is loadable (0 fail-louds, 0 shared-key collisions)\n", seedPath)
|
||||
return nil
|
||||
}
|
||||
|
||||
func report(cfgPath string) error {
|
||||
// Read-only ($0): no API keys required (D20.4 — a store audit must not demand provider keys).
|
||||
r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger())
|
||||
|
|
@ -125,14 +138,14 @@ func report(cfgPath string) error {
|
|||
// extractor / reader-samples read the SAME bytes the backend ships instead of the raw checkpoint. Like
|
||||
// report/status it is $0 and needs no provider keys (D20.4 — an audit surface must not demand keys).
|
||||
// Default output is stable JSON (the machine surface); --plaintext emits the human concatenation.
|
||||
func export(cfgPath string, asPlaintext bool) error {
|
||||
func export(cfgPath string, asPlaintext, asPairs bool) error {
|
||||
r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer r.Close()
|
||||
|
||||
exp, err := r.Export()
|
||||
exp, err := r.Export(asPairs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
|
|||
76
backend/configs/pipeline-arm-deepseek-pro.yaml
Normal file
76
backend/configs/pipeline-arm-deepseek-pro.yaml
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# Свап-арм редактора: DeepSeek-v4-pro reasoning (WS6 / D39.9). Копия боевого C1 (pipeline-c1.yaml) с
|
||||
# ЕДИНСТВЕННЫМ изменением: editor glm-5 → deepseek-v4-pro + few_shot:false. Арм = КОНФИГ (другой
|
||||
# editor-model → другой snapshot_W2). Атрибуция едет в chunk_status.model_actual → export/report.
|
||||
#
|
||||
# ⚠️ ЭХО-МИНА (несущая): deepseek — echo-prone провайдер (thinking-off на CJK → эхо исходника). deepseek-
|
||||
# v4-pro резолвится в ReasoningNone (нет reasoning-capability → reasoning:off = НО-ОП, thinking остаётся
|
||||
# ON) → echoMineViolation НЕ срабатывает, thinking-ON держится (эхо-мина НЕ вооружается). Вход редактора —
|
||||
# русский черновик (не CJK), так что эхо-класс тут вырожден, но thinking-ON держим по дисциплине провайдера.
|
||||
#
|
||||
# few_shot:false (D38.4): у reasoning-редактора собственный CoT нарушается рукописными few-shot-примерами
|
||||
# (exp14 §2а) → ---FEWSHOT---блок editor.md дропается. Фолдится в снапшот (snapshot.go FewShot) — осознанный
|
||||
# --resnapshot. min_max_tokens 8000-флор deepseek-v4-pro (D24.3) держит thinking внутри бюджета.
|
||||
core: C1
|
||||
version: 1
|
||||
|
||||
defaults:
|
||||
max_output_ratio: 2.2
|
||||
min_max_tokens: 2048
|
||||
|
||||
context:
|
||||
glossary_injection: selective
|
||||
glossary_token_budget: 800
|
||||
cache_ttl: "5m"
|
||||
|
||||
segmentation:
|
||||
draft_budget_out: 1797
|
||||
edit_ceiling_out: 3200
|
||||
fertility:
|
||||
cjk: 1.1978
|
||||
other: 0.3852
|
||||
|
||||
retries:
|
||||
regenerate_before_escalate: 1
|
||||
|
||||
stages:
|
||||
- name: draft
|
||||
role: translator
|
||||
model: deepseek-v4-flash
|
||||
prompts:
|
||||
zh-ru: ../prompts/translator.md
|
||||
prompt_version: v1-reflow
|
||||
temperature: 0.3
|
||||
reasoning: "off"
|
||||
escalate_to: deepseek-v4-pro
|
||||
- name: edit
|
||||
role: editor
|
||||
# СВАП-АРМ: deepseek-v4-pro reasoning-редактор (COGS $0.117 vs glm $0.335 WS6(г); thinking-ON держится
|
||||
# через ReasoningNone no-op). Пиннут (editor не эскалирует — D12).
|
||||
model: deepseek-v4-pro
|
||||
prompts:
|
||||
zh-ru: ../prompts/editor.md
|
||||
prompt_version: v3-discourse-reflow
|
||||
temperature: 0.4
|
||||
reasoning: "off" # NO-OP на deepseek-v4-pro (ReasoningNone) → thinking ON; НЕ вооружает эхо-мину
|
||||
few_shot: false # reasoning-редактор: собственный CoT vs рукописные примеры (exp14 §2а) → дроп ---FEWSHOT---
|
||||
|
||||
gates:
|
||||
coverage:
|
||||
enabled: false
|
||||
len_ratio_bounds:
|
||||
zh-ru: [2.2, 4.2]
|
||||
ja-ru: [1.4, 2.6]
|
||||
en-ru: [0.70, 1.4]
|
||||
sent_cov_min: 0.75
|
||||
min_chunk_chars: 500
|
||||
sanitizer:
|
||||
enabled: true
|
||||
|
||||
escalation:
|
||||
chains:
|
||||
default: [deepseek-v4-pro, glm-5.1, gemini-3.1-pro-preview]
|
||||
adult: [grok-4.3]
|
||||
budget_usd: 0
|
||||
|
||||
fanout:
|
||||
candidates: 1
|
||||
74
backend/configs/pipeline-arm-mistral.yaml
Normal file
74
backend/configs/pipeline-arm-mistral.yaml
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Свап-арм редактора: Mistral (WS6 / D39.9 — слабое звено = редактор). Копия боевого C1
|
||||
# (pipeline-c1.yaml) с ЕДИНСТВЕННЫМ изменением: editor.model glm-5 → mistral-large-2512. Арм = КОНФИГ,
|
||||
# не код: другой editor-model → другой stageSnap.Model → другой snapshot_W2 (отдельный прогон, не
|
||||
# конфликтует с базлайном). Атрибуция арма едет в chunk_status.model_actual → export/report.
|
||||
#
|
||||
# ⚠️ ПРЕД-УСЛОВИЕ (WS1 §1б / ревью-2 F4): mistral-large — агрессивный rate-limiter (~48% retry-fails под
|
||||
# N-∥ без guard'а — 00-provider-quirks §Транспорт). models.yaml несёт rate_limit:{max_concurrency:2} на
|
||||
# mistral-large-2512; пер-модельный семафор волнового раннера (WS1b, сдан Block A) реально ограничивает
|
||||
# конкуренцию. Свап-арм mistral НЕ идёт через прод-путь без этого guard.
|
||||
core: C1
|
||||
version: 1
|
||||
|
||||
defaults:
|
||||
max_output_ratio: 2.2
|
||||
min_max_tokens: 2048
|
||||
|
||||
context:
|
||||
glossary_injection: selective
|
||||
glossary_token_budget: 800
|
||||
cache_ttl: "5m"
|
||||
|
||||
# WS2 (слой 1): бюджет нарезки в ВЫХОДНЫХ (ru) токенах — те же ратифицированные zh-ru дефолты, что C1.
|
||||
segmentation:
|
||||
draft_budget_out: 1797
|
||||
edit_ceiling_out: 3200
|
||||
fertility:
|
||||
cjk: 1.1978
|
||||
other: 0.3852
|
||||
|
||||
retries:
|
||||
regenerate_before_escalate: 1
|
||||
|
||||
stages:
|
||||
- name: draft
|
||||
role: translator
|
||||
model: deepseek-v4-flash # черновик — тот же, что базлайн (свап только редактора)
|
||||
prompts:
|
||||
zh-ru: ../prompts/translator.md
|
||||
prompt_version: v1-reflow
|
||||
temperature: 0.3
|
||||
reasoning: "off"
|
||||
escalate_to: deepseek-v4-pro
|
||||
- name: edit
|
||||
role: editor
|
||||
# СВАП-АРМ: mistral-large-2512 вместо glm-5 (D39.9 — редактор = слабое звено; COGS $0.205 vs glm $0.335
|
||||
# WS6(г); не reasoning-модель → эхо-мины нет). Идёт ТОЛЬКО за rate-guard (models.yaml rate_limit).
|
||||
model: mistral-large-2512
|
||||
prompts:
|
||||
zh-ru: ../prompts/editor.md
|
||||
prompt_version: v3-discourse-reflow
|
||||
temperature: 0.4
|
||||
reasoning: "off"
|
||||
# few_shot по дефолту ON: mistral НЕ reasoning-модель, few-shot-примеры P1a-дискурса не мешают её CoT.
|
||||
|
||||
gates:
|
||||
coverage:
|
||||
enabled: false
|
||||
len_ratio_bounds:
|
||||
zh-ru: [2.2, 4.2]
|
||||
ja-ru: [1.4, 2.6]
|
||||
en-ru: [0.70, 1.4]
|
||||
sent_cov_min: 0.75
|
||||
min_chunk_chars: 500
|
||||
sanitizer:
|
||||
enabled: true # премиум-редактор за санитайзером — как C1 (утечки преамбул ловятся)
|
||||
|
||||
escalation:
|
||||
chains:
|
||||
default: [deepseek-v4-pro, glm-5.1, gemini-3.1-pro-preview]
|
||||
adult: [grok-4.3]
|
||||
budget_usd: 0
|
||||
|
||||
fanout:
|
||||
candidates: 1
|
||||
|
|
@ -213,3 +213,63 @@ func TestBoevoyConfigEditorGlm5AndGrokReasoning(t *testing.T) {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSwapArmConfigs guards the WS6 editor swap-arms (D39.9 — редактор = слабое звено). The arms are
|
||||
// CONFIG, not code: each pipeline-arm-*.yaml swaps ONLY the editor model, so a different editor-model
|
||||
// resolves a different stageSnap.Model → a distinct snapshot_W2 (a separate run). This pins: (1) each arm
|
||||
// loads and its editor is the arm model; (2) NEITHER arm re-arms the echo mine — the deepseek-pro editor
|
||||
// resolves to ReasoningNone (thinking ON, off is a no-op) so echoMineViolation stays empty on an echo-
|
||||
// prone provider; (3) the deepseek reasoning-editor drops few-shot (few_shot:false), the mistral arm
|
||||
// keeps it (nil = ON); (4) the mistral arm's PRE-CONDITION — mistral-large-2512 carries a rate_limit so
|
||||
// the WS1 per-model rate-guard actually throttles it (§6(б) F4). All three boevoy pipelines (c1 + two
|
||||
// arms) still pin the same translator+draft; only the editor differs.
|
||||
func TestSwapArmConfigs(t *testing.T) {
|
||||
m, err := LoadModels(filepath.Join("..", "..", "configs", "models.yaml"))
|
||||
if err != nil {
|
||||
t.Fatalf("load boevoy models.yaml: %v", err)
|
||||
}
|
||||
arms := []struct {
|
||||
file, editorModel string
|
||||
fewShotOff bool // the deepseek reasoning editor drops few-shot; mistral keeps it
|
||||
}{
|
||||
{"pipeline-arm-mistral.yaml", "mistral-large-2512", false},
|
||||
{"pipeline-arm-deepseek-pro.yaml", "deepseek-v4-pro", true},
|
||||
}
|
||||
for _, a := range arms {
|
||||
t.Run(a.file, func(t *testing.T) {
|
||||
p, err := LoadPipeline(filepath.Join("..", "..", "configs", a.file), m)
|
||||
if err != nil {
|
||||
t.Fatalf("load %s: %v", a.file, err)
|
||||
}
|
||||
var edit *Stage
|
||||
for i := range p.Stages {
|
||||
if p.Stages[i].Role == "editor" {
|
||||
edit = &p.Stages[i]
|
||||
}
|
||||
}
|
||||
if edit == nil {
|
||||
t.Fatalf("%s has no editor stage", a.file)
|
||||
}
|
||||
if edit.Model != a.editorModel {
|
||||
t.Errorf("%s editor model = %q, want the swap-arm %q", a.file, edit.Model, a.editorModel)
|
||||
}
|
||||
// (2) echo-mine safety: the arm editor's model must not re-arm the mine.
|
||||
if why := m.echoMineViolation(edit.Model); why != "" {
|
||||
t.Errorf("%s editor %q re-arms the echo mine: %s", a.file, edit.Model, why)
|
||||
}
|
||||
// (3) few-shot policy: deepseek reasoning-editor drops the block; mistral keeps it (nil = ON).
|
||||
if a.fewShotOff {
|
||||
if edit.FewShot == nil || *edit.FewShot {
|
||||
t.Errorf("%s deepseek reasoning-editor must set few_shot:false (drop hand examples), got %v", a.file, edit.FewShot)
|
||||
}
|
||||
} else if edit.FewShot != nil {
|
||||
t.Errorf("%s mistral editor must keep few_shot default (nil=ON), got %v", a.file, edit.FewShot)
|
||||
}
|
||||
})
|
||||
}
|
||||
// (4) mistral rate-guard pre-condition: mistral-large-2512 must carry a rate_limit so the WS1 wave
|
||||
// per-model semaphore actually throttles it (mistral ~48% retry-fails under N-∥ without it).
|
||||
if rl := m.Models["mistral-large-2512"].RateLimit; rl.MaxConcurrency <= 0 {
|
||||
t.Errorf("mistral-large-2512 must configure rate_limit.max_concurrency>0 (WS6 pre-condition — the swap-arm needs the WS1 rate-guard), got %d", rl.MaxConcurrency)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -137,6 +137,21 @@ type Gates struct {
|
|||
Glossary GlossaryGate `yaml:"glossary"`
|
||||
Sanitizer SanitizerGate `yaml:"sanitizer"`
|
||||
RegressionGuard RegressionGuardGate `yaml:"regression_guard"`
|
||||
Banknote BanknoteGate `yaml:"banknote"`
|
||||
}
|
||||
|
||||
// BanknoteGate controls the banknote-v1 in-band footnote channel (WS4, RATIFIED D39.10): when enabled,
|
||||
// the translator MAY emit a versioned ⟦TM-BANK-v1⟧ separator + tab-delimited term lines for NEW terms
|
||||
// after the translation — the DIRECT dst delivery the co-occurrence miner could not extract (蛊→гу).
|
||||
// Enabling it is TWO coordinated changes: this backend gate (the runner slices the block off BEFORE the
|
||||
// gates/editor, commits the cleaned draft as a derived export checkpoint, and folds banknoteSnap into the
|
||||
// snapshot) AND the footnote INSTRUCTION baked into the translator prompt file (which moves PromptSHA256).
|
||||
// Opt-in (default false), like the sanitizer/coverage gates; a no-op unless the prompt actually instructs
|
||||
// the model to emit banknotes (a normal draft has no separator → the slice is a no-op). Its parser/slice
|
||||
// VERSION is folded into banknoteSnap (verdict-axis) only when enabled, so a parser change is a loud
|
||||
// --resnapshot even without a prompt edit (§4в point 6).
|
||||
type BanknoteGate struct {
|
||||
Enabled bool `yaml:"enabled"`
|
||||
}
|
||||
|
||||
// RegressionGuardGate controls the post-reflow regression guard (D38 infra-pack,
|
||||
|
|
|
|||
|
|
@ -1,9 +1,15 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"textmachine/backend/internal/config"
|
||||
"textmachine/backend/internal/store"
|
||||
)
|
||||
|
||||
// banknote.go: the banknote-v1 in-band footnote channel (WS4, план §4 / research/20 §B3 / exp16
|
||||
|
|
@ -30,6 +36,12 @@ const bankMaxLines = 12
|
|||
// re-resolves the stripped draft, so it must be a loud --resnapshot even without a prompt edit.
|
||||
const bankParserVersion = "banknote-v1"
|
||||
|
||||
// bankDerivedNS is the derived-checkpoint id NAMESPACE (§4б EXACT formula) — the "tm-<name>-v1" prefix
|
||||
// convention the sibling commitSanitizedExport uses ("tm-sanitized-v1"), kept DISTINCT from the parser
|
||||
// version above exactly as sanitizerSnap.Version (sanitizer-v6) is distinct from the "tm-sanitized-v1"
|
||||
// namespace: it embeds the channel version so a version change yields a fresh (never colliding) id.
|
||||
const bankDerivedNS = "tm-banknote-v1"
|
||||
|
||||
// bankTypeOK is the accepted type set; anything else falls back to "term" (a benign default).
|
||||
var bankTypeOK = map[string]bool{"name": true, "place": true, "title": true, "term": true, "nickname": true}
|
||||
|
||||
|
|
@ -133,3 +145,63 @@ func parseBanknote(block string, truncatedGeneration bool) ([]bankEntry, bankFla
|
|||
flags.ParseFail = bad > 0
|
||||
return entries, flags
|
||||
}
|
||||
|
||||
// --- integration seam (план §4(а) 12 points) ----------------------------------------------------
|
||||
|
||||
// bankTokenBudget is the extra max_tokens the translator draft reserves for the ≤12-line footnote block
|
||||
// (integration point 5): the translation comes FIRST and the block LAST, so a length cut hits the block
|
||||
// (tolerated), not the translation. ~12 tokens per line. Added ONLY when the channel is enabled.
|
||||
const bankTokenBudget = bankMaxLines * 12
|
||||
|
||||
// applyBanknote slices the banknote block off a translator draft (integration points 1–5, 8). It runs
|
||||
// ONLY on the translator role with the channel enabled — the editor never emits banknotes, and a normal
|
||||
// draft has no separator (the slice is then a no-op that returns the raw text unchanged, so a
|
||||
// banknote-OFF run is byte-identical). When a ⟦TM-BANK-v1⟧ block IS present it returns the CLEANED
|
||||
// translation (what classify/coverage/echo/editor/export all see — points 2/3/4) plus that same cleaned
|
||||
// text as the `stripped` export to commit as a derived checkpoint (point 8), and the per-chunk telemetry
|
||||
// (point 10). Candidates are PARSED/accepted only under finish=="stop" (the finish=stop-only gate §4б —
|
||||
// a truncated block is not trusted); the slice itself runs regardless, so the length/echo classify never
|
||||
// sees the footnote's Han src column. Deterministic (a pure function of the raw text + gate state).
|
||||
func (r *Runner) applyBanknote(role, rawText, finish string) (clean, stripped string, flags bankFlags) {
|
||||
if role != roleTranslator || !r.Pipeline.Gates.Banknote.Enabled {
|
||||
return rawText, "", bankFlags{}
|
||||
}
|
||||
cleanText, block := splitBanknote(rawText)
|
||||
if block == "" {
|
||||
return rawText, "", bankFlags{} // no separator → no strip, no derived checkpoint (byte-identical path)
|
||||
}
|
||||
if finish == "stop" {
|
||||
// finish=stop-only gate: accept candidates + count telemetry only for a complete generation.
|
||||
// truncatedGeneration=false is unreachable-otherwise in prod (a truncated gen is finish≠stop).
|
||||
_, flags = parseBanknote(block, false)
|
||||
}
|
||||
return cleanText, cleanText, flags
|
||||
}
|
||||
|
||||
// bankDerivedHash is the CONTENT-ADDRESSED id of a banknote-stripped export checkpoint (§4б EXACT
|
||||
// formula, mirroring commitSanitizedExport's namespacing): sha256("tm-banknote-v1\x00"+reqHash+"\x00"+
|
||||
// stripped), prefixed "tm-banknote-v1:" so it can never collide with a real hex attempt hash and a
|
||||
// resume re-derives the identical id for free. The namespace embeds the channel version → a version
|
||||
// change yields a new id.
|
||||
func bankDerivedHash(reqHash, stripped string) string {
|
||||
sum := sha256.Sum256([]byte(bankDerivedNS + "\x00" + reqHash + "\x00" + stripped))
|
||||
return bankDerivedNS + ":" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// commitBanknoteExport persists the banknote-stripped CLEAN draft as a $0 derived checkpoint and returns
|
||||
// its request_hash (→ chunk_status.final_hash), so the OK-path final_hash→checkpoint export contract
|
||||
// (and the editor's resume read) yields the CLEANED draft instead of the raw one carrying the footnote
|
||||
// (integration point 8). Mirrors commitSanitizedExport: cost 0, escalation 0, idempotent, written BEFORE
|
||||
// chunk_status references it. `att` is the terminal (possibly escalated) OK attempt whose block was
|
||||
// sliced; att.bankStripped is the cleaned text.
|
||||
func (r *Runner) commitBanknoteExport(st config.Stage, ch Chunk, job *store.Job, att stageAttempt) (string, error) {
|
||||
derivedHash := bankDerivedHash(att.reqHash, att.bankStripped)
|
||||
if err := r.Store.PutDerivedCheckpoint(store.Checkpoint{
|
||||
RequestHash: derivedHash, JobID: job.ID, ChunkIdx: ch.ChunkIdx, Attempt: att.attempt,
|
||||
Stage: st.Name, Role: st.Role, ModelRequested: att.modelActual, ModelActual: att.modelActual,
|
||||
ResponseText: att.bankStripped, UsageJSON: "{}", FinishReason: "banknote_export",
|
||||
}); err != nil {
|
||||
return "", fmt.Errorf("pipeline: commit banknote export ch%d/chunk%d/%s: %w", ch.Chapter, ch.ChunkIdx, st.Name, err)
|
||||
}
|
||||
return derivedHash, nil
|
||||
}
|
||||
|
|
|
|||
240
backend/internal/pipeline/banknote_integration_test.go
Normal file
240
backend/internal/pipeline/banknote_integration_test.go
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// banknote_integration_test.go: the WS4 (д) end-to-end pins for the banknote channel wired into the
|
||||
// runner (banknote ON), covering the load-bearing integration points §4(а): the block is SLICED before
|
||||
// the editor/gates (points 1-4), the cleaned draft is committed as a derived export and final_hash is
|
||||
// re-pointed (point 8), resume re-serves the cleaned draft at $0 (point 7), the echo/length classify
|
||||
// runs over the CLEANED text (point 3), the telemetry is persisted (point 10), and banknoteSnap folds
|
||||
// into the snapshot only when enabled (point 6 / §4в). The banknote-OFF golden fixture stays byte-stable
|
||||
// (the TestGolden guard), so these ON-only cells live here rather than in the shared golden.
|
||||
|
||||
// bankBlock is a well-formed 2-line banknote block a translator emits after a clean RU translation.
|
||||
const bankBlock = bankSeparator + "\n龙公\tЛун Гун\tname\n蛊\tгу\tterm"
|
||||
|
||||
func TestBanknoteSliceReTelemetryResume(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
const cleanDraft = "Тихое утро в библиотеке."
|
||||
srv := newJSONProvider(rec, func(body string) (string, string) {
|
||||
if isEditBody(body) {
|
||||
// The editor MUST receive the CLEANED draft — the block was sliced off upstream (point 8).
|
||||
if strings.Contains(body, bankSeparator) || strings.Contains(body, "龙公") {
|
||||
t.Errorf("editor request leaked the banknote block:\n%s", body)
|
||||
}
|
||||
return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop"
|
||||
}
|
||||
return cleanDraft + "\n" + bankBlock, "stop"
|
||||
})
|
||||
defer srv.Close()
|
||||
bookPath := setupProjectOpts(t, srv.URL, projectOpts{banknote: true})
|
||||
ctx := context.Background()
|
||||
|
||||
r1 := newRunner(t, bookPath)
|
||||
res1, err := r1.TranslateBook(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ch := res1.Chunks[0]
|
||||
// The draft stage's usable text is the CLEANED translation (block removed, points 1-3).
|
||||
if got := ch.Stages[0].Text; got != cleanDraft {
|
||||
t.Fatalf("draft text must be the cleaned translation %q, got %q", cleanDraft, got)
|
||||
}
|
||||
if ch.FinalText != "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД" {
|
||||
t.Fatalf("final text must come from the edit over the cleaned draft, got %q", ch.FinalText)
|
||||
}
|
||||
// The draft chunk_status.final_hash must be RE-POINTED at the tm-banknote-v1 derived checkpoint (point 8).
|
||||
cs, _ := r1.Store.GetChunkStatus("test-book", 1, 0, "draft")
|
||||
if cs == nil || !strings.HasPrefix(cs.FinalHash, bankDerivedNS+":") {
|
||||
t.Fatalf("draft final_hash must be a tm-banknote-v1 derived checkpoint, got %+v", cs)
|
||||
}
|
||||
// The derived checkpoint holds the CLEANED draft (so export/resume yield it, not the raw block).
|
||||
dcp, err := r1.Store.GetCheckpoint(cs.FinalHash)
|
||||
if err != nil || dcp == nil {
|
||||
t.Fatalf("derived checkpoint missing: %v", err)
|
||||
}
|
||||
if dcp.ResponseText != cleanDraft || dcp.FinishReason != "banknote_export" {
|
||||
t.Fatalf("derived checkpoint = %q/%q, want cleaned draft / banknote_export", dcp.ResponseText, dcp.FinishReason)
|
||||
}
|
||||
// Telemetry (point 10): 2 accepted lines, no parse fail / truncation.
|
||||
rs, _ := r1.Store.GetRetrievalState("test-book", 1, 0)
|
||||
if rs == nil || rs.NBanknoteLines != 2 || rs.BanknoteParseFail != 0 || rs.BanknoteTruncated != 0 {
|
||||
t.Fatalf("banknote telemetry wrong: %+v", rs)
|
||||
}
|
||||
callsAfterRun1 := rec.count()
|
||||
r1.Close()
|
||||
|
||||
// Resume (new process): $0, NO new provider calls, cleaned draft re-served identically (point 7).
|
||||
r2 := newRunner(t, bookPath)
|
||||
defer r2.Close()
|
||||
res2, err := r2.TranslateBook(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if rec.count() != callsAfterRun1 {
|
||||
t.Fatalf("resume must not re-call the provider, calls %d -> %d", callsAfterRun1, rec.count())
|
||||
}
|
||||
if res2.TotalUSD != 0 || res2.Chunks[0].FinalText != res1.Chunks[0].FinalText {
|
||||
t.Fatalf("resume must be $0 and byte-identical, got %+v", res2.Chunks[0])
|
||||
}
|
||||
}
|
||||
|
||||
// A clean RU translation followed by a Han-heavy banknote block must classify OK — the echo/CJK check
|
||||
// runs over the CLEANED text (point 3). WITHOUT the slice, the block's Han src column would push
|
||||
// cjk_share over the 15% threshold and false-flag cjk_artifact.
|
||||
func TestBanknoteClassifyOverCleaned(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
// A short RU translation + a LONG Han-heavy block: raw cjk_share > 0.15, cleaned cjk_share ≈ 0.
|
||||
bigBlock := bankSeparator + "\n" +
|
||||
strings.Repeat("龙公\tЛун Гун\tname\n古月\tГу Юэ\tname\n方源\tФан Юань\tname\n", 3)
|
||||
srv := newJSONProvider(rec, func(body string) (string, string) {
|
||||
if isEditBody(body) {
|
||||
return "ред", "stop"
|
||||
}
|
||||
return "Он ушёл." + "\n" + bigBlock, "stop"
|
||||
})
|
||||
defer srv.Close()
|
||||
bookPath := setupProjectOpts(t, srv.URL, projectOpts{banknote: true})
|
||||
|
||||
r := newRunner(t, bookPath)
|
||||
defer r.Close()
|
||||
res, err := r.TranslateBook(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ch := res.Chunks[0]
|
||||
if ch.Stages[0].Disposition != DispOK {
|
||||
t.Fatalf("a clean RU draft + Han banknote block must classify OK over the cleaned text, got %+v", ch.Stages[0])
|
||||
}
|
||||
if ch.Stages[0].Text != "Он ушёл." {
|
||||
t.Fatalf("draft text must be the cleaned translation, got %q", ch.Stages[0].Text)
|
||||
}
|
||||
}
|
||||
|
||||
// Enabling the banknote channel folds banknoteSnap into the snapshot (point 6 / §4в); a banknote-OFF
|
||||
// project renders a DIFFERENT snapshot (so a channel flip is a loud --resnapshot). Off omits the field
|
||||
// (omitempty), on adds banknote:{enabled,parser_version}.
|
||||
func TestBanknoteSnapshotFold(t *testing.T) {
|
||||
srv := httptest.NewServer(nil)
|
||||
defer srv.Close()
|
||||
off := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{}))
|
||||
defer off.Close()
|
||||
on := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{banknote: true}))
|
||||
defer on.Close()
|
||||
|
||||
offID, offPayload, err := off.snapshotID()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
onID, onPayload, err := on.snapshotID()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if offID == onID {
|
||||
t.Fatalf("enabling the banknote channel must move the snapshot (loud --resnapshot)")
|
||||
}
|
||||
if strings.Contains(offPayload, "banknote") {
|
||||
t.Fatalf("a banknote-OFF snapshot must omit the field (omitempty), got: %s", offPayload)
|
||||
}
|
||||
if !strings.Contains(onPayload, `"banknote":{"enabled":true,"parser_version":"`+bankParserVersion+`"}`) {
|
||||
t.Fatalf("a banknote-ON snapshot must fold {enabled,parser_version}, got: %s", onPayload)
|
||||
}
|
||||
}
|
||||
|
||||
// A truncated banknote generation (finish=length) is a length flag (retryable) — the block is not parsed
|
||||
// (finish=stop-only gate §4б), and with the tolerant parser held for robustness it never counts a parse
|
||||
// fail. Here a channel-off run is unaffected; this pins that the stop-only gate does not accept a
|
||||
// truncated block's candidates. With regenerate=0 the length draft flags immediately.
|
||||
func TestBanknoteFinishStopOnlyGate(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
srv := newJSONProvider(rec, func(body string) (string, string) {
|
||||
if isEditBody(body) {
|
||||
return "ред", "stop"
|
||||
}
|
||||
// A truncated generation: clean text + a partial block, finish=length.
|
||||
return "Незаконченный перевод" + "\n" + bankSeparator + "\n龙公\tЛун", "length"
|
||||
})
|
||||
defer srv.Close()
|
||||
bookPath := setupProjectOpts(t, srv.URL, projectOpts{banknote: true, regenerate: 0})
|
||||
|
||||
r := newRunner(t, bookPath)
|
||||
defer r.Close()
|
||||
res, err := r.TranslateBook(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ch := res.Chunks[0]
|
||||
// finish=length → flagged length (the block is not accepted; the translation is retried in prod).
|
||||
if ch.Stages[0].Disposition != DispFlagged || ch.Stages[0].FlagReason != FlagLength {
|
||||
t.Fatalf("a truncated generation must flag length (stop-only gate rejects the block), got %+v", ch.Stages[0])
|
||||
}
|
||||
// No candidates accepted under a non-stop finish → zero telemetry.
|
||||
rs, _ := r.Store.GetRetrievalState("test-book", 1, 0)
|
||||
if rs != nil && rs.NBanknoteLines != 0 {
|
||||
t.Fatalf("a non-stop generation must accept 0 banknote lines, got %+v", rs)
|
||||
}
|
||||
}
|
||||
|
||||
// The banknote slice + derived-commit + re-point must run AFTER the escalation resolve and consume the
|
||||
// ESCALATED draft's stripped text (WS4 point 8's escalation-ordering note): the primary echoes CJK
|
||||
// (cjk_artifact) → escalates → the fallback returns a clean RU translation + a banknote block, which is
|
||||
// sliced and committed as a derived checkpoint keyed on the ESCALATED attempt's request_hash.
|
||||
func TestBanknoteEscalatedDraftStripped(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
const fbClean = "Тихое утро в библиотеке."
|
||||
srv := newJSONProvider(rec, func(body string) (string, string) {
|
||||
if isEditBody(body) {
|
||||
if strings.Contains(body, bankSeparator) {
|
||||
t.Errorf("editor leaked the escalated banknote block:\n%s", body)
|
||||
}
|
||||
return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop"
|
||||
}
|
||||
if strings.Contains(body, "fake-fallback") {
|
||||
return fbClean + "\n" + bankBlock, "stop" // the fallback emits a clean translation + banknote
|
||||
}
|
||||
return "静かな図書館の朝。", "stop" // the primary echoes the CJK source → cjk_artifact
|
||||
})
|
||||
defer srv.Close()
|
||||
bookPath := setupEscalationProject(t, srv.URL, 1.0, false, "")
|
||||
pipePath := filepath.Join(filepath.Dir(bookPath), "pipeline.yaml")
|
||||
raw, err := os.ReadFile(pipePath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeFile(t, pipePath, string(raw)+"\ngates:\n banknote:\n enabled: true\n")
|
||||
|
||||
r := newRunner(t, bookPath)
|
||||
defer r.Close()
|
||||
res, err := r.TranslateBook(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ch := res.Chunks[0]
|
||||
draft := ch.Stages[0]
|
||||
if !draft.Escalated || draft.EscalationModel != "fake-fallback" {
|
||||
t.Fatalf("the draft must have escalated to fake-fallback, got %+v", draft)
|
||||
}
|
||||
// The escalated draft's usable text is the CLEANED fallback translation (block sliced).
|
||||
if draft.Text != fbClean {
|
||||
t.Fatalf("escalated draft text must be the cleaned fallback translation %q, got %q", fbClean, draft.Text)
|
||||
}
|
||||
// final_hash re-pointed at a tm-banknote-v1 derived checkpoint holding the cleaned ESCALATED draft.
|
||||
cs, _ := r.Store.GetChunkStatus("test-book", 1, 0, "draft")
|
||||
if cs == nil || !strings.HasPrefix(cs.FinalHash, bankDerivedNS+":") {
|
||||
t.Fatalf("escalated draft final_hash must be a derived checkpoint, got %+v", cs)
|
||||
}
|
||||
dcp, _ := r.Store.GetCheckpoint(cs.FinalHash)
|
||||
if dcp == nil || dcp.ResponseText != fbClean {
|
||||
t.Fatalf("derived checkpoint must hold the cleaned escalated draft, got %+v", dcp)
|
||||
}
|
||||
if ch.FinalText != "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД" {
|
||||
t.Fatalf("final text must come from the edit over the cleaned escalated draft, got %q", ch.FinalText)
|
||||
}
|
||||
}
|
||||
|
|
@ -36,6 +36,9 @@ type StageResult struct {
|
|||
// fallback passed the re-gate, Model above is the fallback (it answered).
|
||||
Escalated bool
|
||||
EscalationModel string // the fallback model when Escalated ("" otherwise)
|
||||
// BankFlags carries the translator draft's banknote telemetry (WS4 point 10): accepted line count +
|
||||
// parse-fail / truncation flags. Zero-valued on every non-translator stage and every channel-off run.
|
||||
BankFlags bankFlags
|
||||
}
|
||||
|
||||
// ChunkOutcome is one chunk's result across the stage list.
|
||||
|
|
@ -142,22 +145,23 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
|
|||
"chapters", len(doc.Chapters), "chunks", len(chunks), "stages_per_chunk", len(r.Pipeline.Stages))
|
||||
|
||||
res := &BookResult{BookID: r.Book.BookID}
|
||||
// Sticky scene-inertia state (A5): the exact-matched ids of the recent chunks in
|
||||
// the CURRENT chapter, reset at a chapter boundary (a new chapter is a scene change).
|
||||
// Rebuilt deterministically each run because translateChunk recomputes the ($0)
|
||||
// selection for EVERY chunk, resumed ones included — so sticky is resume-stable.
|
||||
var stickyWin []map[string]injectionDisposition
|
||||
// W0: pre-compute the sticky-chain memory selection for EVERY chunk (WS1 §1б, precomputeSticky). The
|
||||
// sticky window (A5 scene-inertia — the recent chunks' exact matches in the current chapter, reset at
|
||||
// a chapter boundary) is a CROSS-CHUNK sequential dependency, so it is computed once here in W0 and
|
||||
// consumed per-chunk below. This is byte-identical to the old inline computation (Select is $0 and
|
||||
// pure; the golden test proves the injected bytes did not move) — and it is the exact precompute the
|
||||
// parallel wave executor will consume (the cross-chunk state cannot be recomputed inside a wave).
|
||||
stickySel := precomputeSticky(chunks, r.memory, r.Pipeline.Context.GlossaryTokenBudget)
|
||||
prevChapter := 0
|
||||
for i, ch := range chunks {
|
||||
if ch.Chapter != prevChapter {
|
||||
stickyWin = nil
|
||||
prevChapter = ch.Chapter
|
||||
// Пульс прогресса на границе главы: done/total + деньги ЭТОГО прогона —
|
||||
// оператор длинной книги видит движение, не открывая status.
|
||||
r.Log.InfoContext(ctx, "chapter started", "chapter", ch.Chapter,
|
||||
"chunks_done", i, "chunks_total", len(chunks), "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
||||
}
|
||||
outcome, activeIDs, err := r.translateChunk(ctx, snapID, ch, unionSticky(stickyWin))
|
||||
outcome, err := r.translateChunk(ctx, snapID, ch, stickySel[i])
|
||||
if err != nil {
|
||||
// Infra failure: abort. Chunks already committed are checkpointed;
|
||||
// resume continues from here at $0 for the done work.
|
||||
|
|
@ -168,11 +172,6 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
|
|||
if outcome.Disposition == DispFlagged {
|
||||
res.Flagged++
|
||||
}
|
||||
// Advance the sticky window (keep the last stickyDepth chunks' exact matches).
|
||||
stickyWin = append(stickyWin, activeIDs)
|
||||
if len(stickyWin) > stickyDepth {
|
||||
stickyWin = stickyWin[len(stickyWin)-stickyDepth:]
|
||||
}
|
||||
}
|
||||
r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID,
|
||||
"chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
||||
|
|
|
|||
|
|
@ -38,7 +38,12 @@ import (
|
|||
// false-flagged against a STRAY output integer (a year, a count) — only a mismatching magnitude
|
||||
// WORD triggers, bare integers can merely confirm; (3) same mechanism covers a 万-in-a-name +
|
||||
// unrelated output number. No live book has run under v1 (D18 acceptance pending), so the re-pin is free.
|
||||
const cheapGateVersion = "cheapgate-v2"
|
||||
//
|
||||
// v3 (WS5, R4) adds the defect-class checkers DC1 (时辰 double-hour units), DC2 (千万/数十万 magnitude
|
||||
// scale) and DC6 (register negative-list — the zh-ru pack, checkers_zh_ru.go). They are observability
|
||||
// (never a disposition), tuned precision-over-recall; the bump is a loud --resnapshot as the discipline
|
||||
// requires (a rule/pack edit shifts recorded counts). They fire 0 on a non-zh source / non-register text.
|
||||
const cheapGateVersion = "cheapgate-v3-dc-checkers"
|
||||
|
||||
// cheapGateConfig carries the brief-derived knobs: the ё-policy and the per-project allowlist of
|
||||
// surfaces that look like a blocklisted interjection but are legitimate here (e.g. a character
|
||||
|
|
@ -64,11 +69,17 @@ type cheapGateResult struct {
|
|||
// cfg.regressionEnabled, so a book that does not enable the guard serialises identically.
|
||||
LengthCollapse int `json:"length_collapse,omitempty"`
|
||||
NumberDrift int `json:"number_drift,omitempty"`
|
||||
// DC1TimeUnits / DC2Magnitude / DC6Register are the WS5 defect-class checkers (checkers_zh_ru.go),
|
||||
// observability like the others. Zero on a non-zh source / non-register final (the golden fixture).
|
||||
DC1TimeUnits int `json:"dc1_time_units,omitempty"`
|
||||
DC2Magnitude int `json:"dc2_magnitude,omitempty"`
|
||||
DC6Register int `json:"dc6_register,omitempty"`
|
||||
Detail []string `json:"detail,omitempty"`
|
||||
}
|
||||
|
||||
func (c cheapGateResult) total() int {
|
||||
return c.DialogueDash + c.YoInconsistent + c.TranslitInterj + c.NumberMagnitude + c.LengthCollapse + c.NumberDrift
|
||||
return c.DialogueDash + c.YoInconsistent + c.TranslitInterj + c.NumberMagnitude + c.LengthCollapse + c.NumberDrift +
|
||||
c.DC1TimeUnits + c.DC2Magnitude + c.DC6Register
|
||||
}
|
||||
|
||||
// runCheapGates runs the four always-on style flaggers over one chunk's source and FINAL text, plus
|
||||
|
|
@ -87,6 +98,16 @@ func runCheapGates(source, draft, final string, cfg cheapGateConfig) cheapGateRe
|
|||
n, det = lintNumberMagnitude(source, final)
|
||||
r.NumberMagnitude = n
|
||||
r.Detail = append(r.Detail, det...)
|
||||
// WS5 defect-class checkers (DC1/DC2/DC6, checkers_zh_ru.go) — src↔target observability flaggers.
|
||||
n, det = lintTimeUnits(source, final)
|
||||
r.DC1TimeUnits = n
|
||||
r.Detail = append(r.Detail, det...)
|
||||
n, det = lintMagnitudeScale(source, final)
|
||||
r.DC2Magnitude = n
|
||||
r.Detail = append(r.Detail, det...)
|
||||
n, det = lintRegisterLexicon(final)
|
||||
r.DC6Register = n
|
||||
r.Detail = append(r.Detail, det...)
|
||||
if cfg.regressionEnabled {
|
||||
rg := runRegressionGuard(draft, final)
|
||||
r.LengthCollapse = rg.LengthCollapse
|
||||
|
|
@ -210,7 +231,7 @@ func quoteShape(after []rune) bool {
|
|||
// needs a ё-dictionary (deferred, B-tier). Names — the primary target (Пётр/Петр) — are never
|
||||
// homographs, so they are always caught regardless.
|
||||
var yoHomographEForms = map[string]bool{
|
||||
"все": true, "всех": true, "всем": true, "всеми":true,
|
||||
"все": true, "всех": true, "всем": true, "всеми": true,
|
||||
"небо": true, "небом": true,
|
||||
"узнаем": true, "узнаете": true, "узнает": true,
|
||||
"падеж": true, "падежа": true,
|
||||
|
|
|
|||
168
backend/internal/pipeline/checkers_zh_ru.go
Normal file
168
backend/internal/pipeline/checkers_zh_ru.go
Normal file
|
|
@ -0,0 +1,168 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// checkers_zh_ru.go: the WS5 defect-class checkers DC1 (时辰 double-hour units), DC2 (千万/数十万 magnitude
|
||||
// scale) and DC6 (register negative-list) — deterministic, $0 flaggers on the source↔FINAL text, ported
|
||||
// from the frozen ws5_checkers_verify.py. Like the four cheap style gates they are OBSERVABILITY, NEVER a
|
||||
// disposition (research/20: judges too noisy → deterministic gates; a hit is recorded in the retrieval-
|
||||
// state / report, never dropping a chunk). They are tuned PRECISION over recall — DC1/DC2 fire only on an
|
||||
// EXPLICIT src↔target mismatch; the LANDING (whether the flag is trusted) is gated by a §5(д) false-
|
||||
// positive measure on a fresh rerun, not by this build. Their rule VERSION rides cheapGateVersion (folded
|
||||
// into the snapshot as StyleCheckVersion), so editing a rule / the pack is a loud --resnapshot.
|
||||
//
|
||||
// PER-PAIR PACK (слой-2 data, §5(в)): the unit table + negative-list below are the zh-ru convention pack.
|
||||
// They are code consts versioned by cheapGateVersion (the same discipline the existing cheap-gate data
|
||||
// uses), not a config-loaded per-pair file — a config-loaded versioned pack is a cleaner future form
|
||||
// (noted residual). The checkers self-gate on source content (时辰/千万 are zh-specific → no fire on a
|
||||
// non-zh source), and DC6 is target-side (gated behind ru like the other readability flaggers).
|
||||
|
||||
// --- DC-1: 时辰 (double-hour) unit checker (ws5.shichen_checker) -----------------------------------
|
||||
|
||||
// dcCNNum maps the small CJK numerals the 时辰 pattern accepts to their value (1 时辰 = 2 modern hours).
|
||||
var dcCNNum = map[rune]int{
|
||||
'一': 1, '二': 2, '两': 2, '三': 3, '四': 4, '五': 5, '六': 6, '七': 7, '八': 8, '九': 9, '十': 10,
|
||||
}
|
||||
|
||||
// dcShichenRE captures the count before 时辰 (an Arabic digit or a small CJK numeral), tolerating 个.
|
||||
var dcShichenRE = regexp.MustCompile(`([0-9一二两三四五六七八九十])\s*个?\s*时辰`)
|
||||
|
||||
// dcRuHoursRE captures a Russian "<count> час…" rendering; the alternatives mirror the reference map.
|
||||
var dcRuHoursRE = regexp.MustCompile(`(\d+|один|два|двух|три|трёх|трех|четыре|пять|шесть)\s+час`)
|
||||
|
||||
// dcRuHourWord maps the Russian count words dcRuHoursRE captures to their value.
|
||||
var dcRuHourWord = map[string]int{
|
||||
"один": 1, "два": 2, "двух": 2, "три": 3, "трёх": 3, "трех": 3, "четыре": 4, "пять": 5, "шесть": 6,
|
||||
}
|
||||
|
||||
// lintTimeUnits flags a 时辰 (=2h) unit error: N个时辰 rendered as N часов (the count copied as hours)
|
||||
// instead of ~2N hours (三个时辰 → «три часа» should be ~6h). It fires ONLY on an explicit mismatch — a
|
||||
// paraphrase with no hours count is a valid rendering, not a defect (the reference dropped the "no hours
|
||||
// found" branch that false-flagged 1.8%). Pure and deterministic.
|
||||
func lintTimeUnits(source, final string) (int, []string) {
|
||||
m := dcShichenRE.FindStringSubmatch(source)
|
||||
if m == nil {
|
||||
return 0, nil
|
||||
}
|
||||
n, ok := dcParseCount(m[1])
|
||||
if !ok {
|
||||
return 0, nil
|
||||
}
|
||||
hm := dcRuHoursRE.FindStringSubmatch(final)
|
||||
if hm == nil {
|
||||
return 0, nil // no explicit hours rendering → a valid paraphrase, not a defect
|
||||
}
|
||||
ruNum, ok := dcParseRuHours(hm[1])
|
||||
if !ok {
|
||||
return 0, nil
|
||||
}
|
||||
expectedHours := n * 2
|
||||
if ruNum == n && ruNum != expectedHours {
|
||||
return 1, []string{fmt.Sprintf("DC1 时辰: %d个时辰 передано как «%d час…» (счёт как часы) вместо ~%d ч (1 时辰 = 2 ч)", n, ruNum, expectedHours)}
|
||||
}
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// dcParseCount parses the DC1 count group: an Arabic digit string or a single small CJK numeral.
|
||||
func dcParseCount(s string) (int, bool) {
|
||||
if v, err := strconv.Atoi(s); err == nil {
|
||||
return v, true
|
||||
}
|
||||
r := []rune(s)
|
||||
if len(r) == 1 {
|
||||
if v, ok := dcCNNum[r[0]]; ok {
|
||||
return v, true
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// dcParseRuHours parses the DC1 hours group: an Arabic digit string or a Russian count word.
|
||||
func dcParseRuHours(s string) (int, bool) {
|
||||
if v, err := strconv.Atoi(s); err == nil {
|
||||
return v, true
|
||||
}
|
||||
if v, ok := dcRuHourWord[s]; ok {
|
||||
return v, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// --- DC-2: number-scale magnitude checker (ws5.magnitude_checker, 千万 / 数十万) --------------------
|
||||
|
||||
// DC2 regexes — a BYTE-FAITHFUL port of ws5.magnitude_checker (_MAG). The ok_re SUPPRESSION guards are
|
||||
// case-INsensitive (the reference passes re.I); the FIRE predicates are case-SENSITIVE (the reference
|
||||
// does NOT pass re.I to the inner тысяч/десятки-тысяч searches). \w is ported as \p{L}* (RE2's \w is
|
||||
// ASCII; the Cyrillic word-continuation the reference intends is letters).
|
||||
var (
|
||||
dcQianwanOkRE = regexp.MustCompile(`(?i)десят\p{L}* миллион|10\s*000\s*000|10000000`) // 千万 correct render → suppress
|
||||
dcShushiwanOkRE = regexp.MustCompile(`(?i)сотн\p{L}* тысяч|нескольк\p{L}* сот\p{L}* тысяч|[1-9]00\s*000`) // 数十万 correct → suppress
|
||||
dcDesyatkiTysRE = regexp.MustCompile(`десятк\p{L}* тысяч`) // 数十万 under-render (case-SENSITIVE, per reference)
|
||||
)
|
||||
|
||||
// lintMagnitudeScale flags a 千万 (10^7) / 数十万 (~several×10^5) magnitude rendered at a WRONG smaller
|
||||
// scale (a разряд error DC2 targets). 千万 → «тысячи» without «миллион» = 10000× under; 数十万 → «десятки
|
||||
// тысяч» = 10× under. A CORRECT rendering anywhere in the chunk (the ok_re guard) SUPPRESSES the flag, so
|
||||
// a chunk that renders 数十万 as «сотни тысяч» does not fire even if «десятки тысяч» appears elsewhere as
|
||||
// an unrelated quantity (the ws5 reference parity — required so the §5(д) FP-measure taken against the
|
||||
// frozen reference matches what ships). ⚠ 千万 is also a stock HYPERBOLE («несметно») whose «тысячи»
|
||||
// rendering is in-register literary, NOT a hard error (§5 A4) — hyperbole-exposed, landing FP-gated §5(д).
|
||||
// Observability only. (Word↔word fraction inversion — 四成四=44% via q4a rule_b1/b2 — is a residual
|
||||
// sub-checker, not built here.)
|
||||
func lintMagnitudeScale(source, final string) (int, []string) {
|
||||
var flags []string
|
||||
if strings.Contains(source, "千万") && !dcQianwanOkRE.MatchString(final) {
|
||||
if strings.Contains(final, "тысяч") && !strings.Contains(final, "миллион") { // case-sensitive, per reference
|
||||
flags = append(flags, "DC2 千万=10^7 передано как «тысячи» (≈10000× занижение) — возможна ошибка разряда (гипербола-риск, §5-A4)")
|
||||
}
|
||||
}
|
||||
if strings.Contains(source, "数十万") && !dcShushiwanOkRE.MatchString(final) {
|
||||
if dcDesyatkiTysRE.MatchString(final) {
|
||||
flags = append(flags, "DC2 数十万≈неск.×10^5 передано как «десятки тысяч» (≈10× занижение)")
|
||||
}
|
||||
}
|
||||
return len(flags), flags
|
||||
}
|
||||
|
||||
// --- DC-6: register-lexicon negative-list (ws5.register_checker) ----------------------------------
|
||||
|
||||
// dcRegisterNegList is the zh-ru register negative-list (слой-2 pack): fairy-tale-Russian / chancery
|
||||
// lexemes that break the xianxia register («терем» in a cultivation novel is a hard register error). The
|
||||
// owner extends this per corpus finding. Lower-cased; whole-word matched.
|
||||
var dcRegisterNegList = []string{"терем", "терема", "тереме", "теремом", "терему", "теремах"}
|
||||
|
||||
// lintRegisterLexicon flags whole-word occurrences of a register negative-list lexeme in the FINAL text.
|
||||
func lintRegisterLexicon(final string) (int, []string) {
|
||||
low := []rune(strings.ToLower(final))
|
||||
hitSet := map[string]bool{}
|
||||
for _, w := range dcRegisterNegList {
|
||||
wr := []rune(w)
|
||||
for i := 0; i+len(wr) <= len(low); i++ {
|
||||
if !runesEqual(low[i:i+len(wr)], wr) {
|
||||
continue
|
||||
}
|
||||
if (i == 0 || !isCyrLetter(low[i-1])) && (i+len(wr) == len(low) || !isCyrLetter(low[i+len(wr)])) {
|
||||
hitSet[w] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(hitSet) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
hits := make([]string, 0, len(hitSet))
|
||||
for w := range hitSet {
|
||||
hits = append(hits, w)
|
||||
}
|
||||
sort.Strings(hits)
|
||||
return len(hits), []string{"DC6 регистр: сказочно-русская лексика вне жанра сянься: " + strings.Join(hits, ", ")}
|
||||
}
|
||||
|
||||
// isCyrLetter reports whether r is a Cyrillic letter (the word boundary for the register match).
|
||||
func isCyrLetter(r rune) bool { return unicode.IsLetter(r) && unicode.Is(unicode.Cyrillic, r) }
|
||||
107
backend/internal/pipeline/checkers_zh_ru_test.go
Normal file
107
backend/internal/pipeline/checkers_zh_ru_test.go
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// checkers_zh_ru_test.go: WS5 (г) — the DC1/DC2/DC6 checkers must CATCH the empirical trap positives
|
||||
// (ws5_checkers_verify.py §a) and stay SILENT on clean/in-register text (precision over recall). These
|
||||
// are the regression fixtures the plan requires (positives from exp15 §7.9 / exp14b).
|
||||
|
||||
func TestDC1TimeUnits(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, src, tgt string
|
||||
wantFlag bool
|
||||
}{
|
||||
{"count-as-hours", "他闭关了三个时辰。", "Он затворился на три часа.", true}, // 三时辰=6h rendered «три часа»
|
||||
{"correct-conversion", "他闭关了三个时辰。", "Он затворился на шесть часов.", false}, // 6h — correct, no flag
|
||||
{"paraphrase-no-hours", "他闭关了三个时辰。", "Он затворился надолго.", false}, // no hours count → valid
|
||||
{"no-shichen", "他走了三里。", "Он прошёл три ли.", false},
|
||||
{"arabic-count", "过了2个时辰。", "Прошло 2 часа.", true}, // 2时辰=4h rendered «2 часа»
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
n, _ := lintTimeUnits(c.src, c.tgt)
|
||||
if (n > 0) != c.wantFlag {
|
||||
t.Fatalf("lintTimeUnits(%q,%q) fired=%v, want %v", c.src, c.tgt, n > 0, c.wantFlag)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDC2MagnitudeScale(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, src, tgt string
|
||||
wantFlag bool
|
||||
}{
|
||||
{"qianwan-as-thousands", "有千万条蛊虫。", "Там были тысячи гу-червей.", true}, // 千万=10M as «тысячи»
|
||||
{"qianwan-correct", "有千万条蛊虫。", "Там были десятки миллионов гу-червей.", false}, // «миллион» present → ok
|
||||
{"shushiwan-as-tens", "聚集了数十万人。", "Собрались десятки тысяч человек.", true}, // 数十万 as «десятки тысяч»
|
||||
{"shushiwan-correct", "聚集了数十万人。", "Собрались сотни тысяч человек.", false}, // «сотни тысяч» → ok
|
||||
// ws5 parity (R4-review MAJOR): a CORRECT rendering «сотни тысяч» SUPPRESSES the flag even when
|
||||
// «десятки тысяч» appears elsewhere as an unrelated quantity (the ok_re guard).
|
||||
{"shushiwan-okre-suppress", "聚集了数十万人。", "Здесь сотни тысяч воинов, а не десятки тысяч слуг.", false},
|
||||
// ws5 parity (R4-review MINOR): the inner fire check is CASE-SENSITIVE (reference no re.I), so a
|
||||
// sentence-initial capitalized «Десятки тысяч» does NOT fire.
|
||||
{"case-sensitive-capitalized", "聚集了数十万人。", "Десятки тысяч человек собрались.", false},
|
||||
{"no-magnitude", "他有三个朋友。", "У него три друга.", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
n, _ := lintMagnitudeScale(c.src, c.tgt)
|
||||
if (n > 0) != c.wantFlag {
|
||||
t.Fatalf("lintMagnitudeScale(%q,%q) fired=%v, want %v", c.src, c.tgt, n > 0, c.wantFlag)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDC6RegisterLexicon(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, tgt string
|
||||
wantN int
|
||||
}{
|
||||
{"terem", "Он вошёл в высокий терем.", 1},
|
||||
{"terem-inflected", "В тереме было тихо, у терема стоял страж.", 2}, // тереме + терема → 2 distinct register forms
|
||||
{"clean", "Он вошёл в высокий зал павильона.", 0},
|
||||
{"substring-not-word", "Термин был странным.", 0}, // «терм» inside «термин» must NOT fire (whole-word)
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
n, det := lintRegisterLexicon(c.tgt)
|
||||
if n != c.wantN {
|
||||
t.Fatalf("lintRegisterLexicon(%q) = %d (%v), want %d", c.tgt, n, det, c.wantN)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The DC checkers must stay SILENT on a non-zh / clean chunk (they self-gate on content), so a
|
||||
// non-Chinese book never accrues style flags from them.
|
||||
func TestDCCheckersSilentOnCleanChunk(t *testing.T) {
|
||||
r := runCheapGates("静かな図書館の朝。", "", "Тихое утро в библиотеке.", cheapGateConfig{})
|
||||
if r.DC1TimeUnits != 0 || r.DC2Magnitude != 0 || r.DC6Register != 0 {
|
||||
t.Fatalf("DC checkers must be silent on a clean ja→ru chunk, got %+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDC3GenderInjection pins the DC3 gender constraint annotation on the editor block (WS5 §5(б)): a
|
||||
// gendered CONFIRMED term carries a gender directive; a genderless term does not (byte-identical line).
|
||||
func TestDC3GenderInjection(t *testing.T) {
|
||||
mk := func(gender string) []pickedEntry {
|
||||
return []pickedEntry{{entry: &memoryEntry{src: "方源", dst: "Фан Юань", status: "approved", gender: gender}, via: "方源", disp: memConfirmed}}
|
||||
}
|
||||
male := renderEditorConstraintBlock(mk("male"))
|
||||
if !strings.Contains(male, "方源 → «Фан Юань» (муж.") {
|
||||
t.Fatalf("male term must carry a masculine directive, got:\n%s", male)
|
||||
}
|
||||
hidden := renderEditorConstraintBlock(mk("hidden"))
|
||||
if !strings.Contains(hidden, "пол СКРЫТ") {
|
||||
t.Fatalf("hidden term must carry the gender-avoidance mandate, got:\n%s", hidden)
|
||||
}
|
||||
none := renderEditorConstraintBlock(mk(""))
|
||||
if !strings.HasSuffix(none, "«Фан Юань»") {
|
||||
t.Fatalf("a genderless term must have NO gender note (line ends at the dst), got:\n%s", none)
|
||||
}
|
||||
}
|
||||
|
|
@ -87,7 +87,7 @@ func (r *Runner) classifyOutput(role, source, output, finish string, isFinal boo
|
|||
// persists the per-chunk retrieval-state (observability). Returns the chunk outcome, the
|
||||
// exact-matched entity ids (the next chunk's sticky_prev, A5), and an error only on an
|
||||
// infra failure. stickyPrev is the prior chunks' exact matches in this chapter.
|
||||
func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, stickyPrev map[string]injectionDisposition) (*ChunkOutcome, map[string]injectionDisposition, error) {
|
||||
func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, memSel memorySelection) (*ChunkOutcome, error) {
|
||||
out := &ChunkOutcome{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: DispOK}
|
||||
prev := ""
|
||||
flagged := false
|
||||
|
|
@ -100,19 +100,20 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st
|
|||
// draftText is the FIRST (translator) stage's output — the pre-reflow baseline for the opt-in
|
||||
// regression guard's draft→final comparison (== final in a single-stage pipeline).
|
||||
draftText := ""
|
||||
// bankTelem carries the translator draft's banknote telemetry (WS4 point 10) into the per-chunk
|
||||
// retrieval-state. Zero for every channel-off / non-banknote chunk.
|
||||
var bankTelem bankFlags
|
||||
|
||||
// Hot path: select the glossary records for this chunk ONCE (deterministic, $0), and serialize the
|
||||
// per-role injection blocks via the role→renderer registry (D39 слой 7). Recomputed on every run
|
||||
// (resumed chunks included) so the sticky window and the injected bytes are resume-stable.
|
||||
var memSel memorySelection
|
||||
// Hot path: the glossary selection for this chunk is PRE-COMPUTED in W0 (precomputeSticky, WS1 §1б)
|
||||
// and passed in — the sticky chain is a cross-chunk dependency the parallel waves cannot recompute.
|
||||
// It is a pure deterministic function of (chunk, sticky_prev, frozen bank), so a resumed chunk
|
||||
// reproduces the identical injected bytes. Serialize the per-role injection blocks via the
|
||||
// role→renderer registry (D39 слой 7).
|
||||
injectionByRole := map[string]string{}
|
||||
activeIDs := map[string]injectionDisposition{}
|
||||
if r.memory != nil {
|
||||
memSel = r.memory.Select(ch.Text, ch.Chapter, stickyPrev, r.Pipeline.Context.GlossaryTokenBudget)
|
||||
for role, render := range roleInjectionRenderers {
|
||||
injectionByRole[role] = render(memSel.injected) // pure → per-role result is map-order-independent
|
||||
}
|
||||
activeIDs = memSel.activeIDs
|
||||
// The disposition-gated suppressor refused to let a lower-trust longer key eat a higher-trust
|
||||
// nested one (D39 слой 4): the approved term survives, but surface the seed-hygiene collision
|
||||
// loudly so an operator can reconcile the draft-nesting-over-approved seed (research/13 §7).
|
||||
|
|
@ -132,7 +133,7 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st
|
|||
SnapshotID: snapID, Disposition: string(DispSkipped), FlagReason: string(flagReason),
|
||||
Detail: detail,
|
||||
}); err != nil {
|
||||
return out, activeIDs, fmt.Errorf("pipeline: record skipped chunk_status: %w", err)
|
||||
return out, fmt.Errorf("pipeline: record skipped chunk_status: %w", err)
|
||||
}
|
||||
out.Stages = append(out.Stages, StageResult{
|
||||
Stage: st.Name, Role: st.Role, Model: st.Model,
|
||||
|
|
@ -149,10 +150,13 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st
|
|||
injection := injectionByRole[st.Role]
|
||||
sr, err := r.runStage(ctx, st, stageIdx, snapID, ch, prev, injection)
|
||||
if err != nil {
|
||||
return out, activeIDs, err
|
||||
return out, err
|
||||
}
|
||||
out.Stages = append(out.Stages, *sr)
|
||||
out.CostUSD += sr.CostUSD
|
||||
if st.Role == roleTranslator {
|
||||
bankTelem = sr.BankFlags // translator draft's banknote telemetry (WS4 point 10)
|
||||
}
|
||||
if sr.Disposition == DispFlagged {
|
||||
flagged = true
|
||||
flagReason = sr.FlagReason
|
||||
|
|
@ -211,8 +215,8 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st
|
|||
// idempotent, so a resume re-derives the identical row. Only when a glossary is materialized
|
||||
// (always true in a normal run — seedGlossary sets an at-least-empty bank).
|
||||
if r.memory != nil {
|
||||
if err := r.persistRetrievalState(snapID, ch, memSel, postMisses, outputChecked, cheap); err != nil {
|
||||
return out, activeIDs, err
|
||||
if err := r.persistRetrievalState(snapID, ch, memSel, postMisses, outputChecked, cheap, bankTelem); err != nil {
|
||||
return out, err
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -229,7 +233,7 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st
|
|||
} else {
|
||||
out.FinalText = exportNormalize(prev)
|
||||
}
|
||||
return out, activeIDs, nil
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// injectionRenderer serializes a chunk's selected memory records into a role's injection message.
|
||||
|
|
@ -249,10 +253,13 @@ var roleInjectionRenderers = map[string]injectionRenderer{
|
|||
// selection + the post-check result. n_exact_hits/n_sticky/n_ambiguous count the INJECTED
|
||||
// records (what the model saw); spoiler/eviction are the dropped-and-logged totals;
|
||||
// post-check misses are recorded only when the translator actually produced text.
|
||||
func (r *Runner) persistRetrievalState(snapID string, ch Chunk, sel memorySelection, misses []postcheckMiss, outputChecked bool, cheap cheapGateResult) error {
|
||||
func (r *Runner) persistRetrievalState(snapID string, ch Chunk, sel memorySelection, misses []postcheckMiss, outputChecked bool, cheap cheapGateResult, bank bankFlags) error {
|
||||
rs := store.RetrievalState{
|
||||
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, SnapshotID: snapID,
|
||||
NStyleFlags: cheap.total(),
|
||||
NBanknoteLines: bank.NLines,
|
||||
BanknoteParseFail: boolToStoreInt(bank.ParseFail),
|
||||
BanknoteTruncated: boolToStoreInt(bank.Truncated),
|
||||
}
|
||||
if cheap.total() > 0 {
|
||||
if b, err := json.Marshal(cheap); err == nil {
|
||||
|
|
@ -304,6 +311,14 @@ func (r *Runner) persistRetrievalState(snapID string, ch Chunk, sel memorySelect
|
|||
return r.Store.UpsertRetrievalState(rs)
|
||||
}
|
||||
|
||||
// boolToStoreInt maps a bool telemetry flag to the store's 0/1 integer column form.
|
||||
func boolToStoreInt(b bool) int {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// cheapGateConfig builds the cheap-gate knobs from the book brief: the ё-policy and the
|
||||
// lower-cased per-project interjection allowlist. Pure, no store access.
|
||||
func (r *Runner) cheapGateConfig() cheapGateConfig {
|
||||
|
|
|
|||
|
|
@ -35,6 +35,11 @@ type ChunkExport struct {
|
|||
// substantive flag / upstream-skip / pending chunk (exportNormalize("")=="", so the paths coincide
|
||||
// byte-for-byte with ChunkOutcome.FinalText).
|
||||
FinalText string `json:"final_text"`
|
||||
// Source is the chunk's normalized source text — populated ONLY under `tmctl export --pairs` (WS5
|
||||
// §5(д) / research/20 §E-3), the src↔target column the polygon needs to measure the DC1 (时辰) / DC2
|
||||
// (千万) source-vs-target checkers' false-positive rate. Omitted by default (the export contract is
|
||||
// target-only); the source is the $0 manifest re-chunk, never a stored/billed artifact.
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
// BookExport is the whole-book export projection (chunks in manifest — (chapter, chunk_idx) — order, so
|
||||
|
|
@ -70,7 +75,7 @@ type BookExport struct {
|
|||
// - MANIFEST (F4): a chunk absent from the current source is a GHOST row (dropped + counted + WARNed);
|
||||
// a manifest chunk with no final row is PENDING; both keep a partial/edited book from exporting as
|
||||
// if complete (the D37-skew the polygon extractor must not inherit).
|
||||
func (r *Runner) Export() (*BookExport, error) {
|
||||
func (r *Runner) Export(pairs bool) (*BookExport, error) {
|
||||
statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -124,14 +129,21 @@ func (r *Runner) Export() (*BookExport, error) {
|
|||
exp.TotalChunks++
|
||||
cs, ok := finalRow[k]
|
||||
if !ok {
|
||||
pend := ChunkExport{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: exportPending}
|
||||
if pairs {
|
||||
pend.Source = ch.Text
|
||||
}
|
||||
exp.PendingChunks++
|
||||
exp.Chunks = append(exp.Chunks, ChunkExport{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: exportPending})
|
||||
exp.Chunks = append(exp.Chunks, pend)
|
||||
continue
|
||||
}
|
||||
ce, err := r.chunkExport(cs, gateOn, missByChunk[k])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if pairs {
|
||||
ce.Source = ch.Text // the src↔target column for the DC1/DC2 FP-measure (--pairs)
|
||||
}
|
||||
exp.Chunks = append(exp.Chunks, ce)
|
||||
}
|
||||
// F4: GHOST rows — a stored final row whose chunk is NOT in the current manifest (source shrunk since
|
||||
|
|
|
|||
61
backend/internal/pipeline/export_pairs_test.go
Normal file
61
backend/internal/pipeline/export_pairs_test.go
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// export_pairs_test.go: WS5 `tmctl export --pairs` — the src↔target column the polygon needs for the
|
||||
// DC1/DC2 false-positive measure (§5(д)). Default export is target-only (Source omitted); --pairs adds
|
||||
// the $0 manifest source per chunk, matching what the runner chunked.
|
||||
func TestExportPairsIncludesSource(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
srv := newJSONProvider(rec, draftEdit)
|
||||
defer srv.Close()
|
||||
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "静かな図書館の朝。"})
|
||||
ctx := context.Background()
|
||||
|
||||
r := newRunner(t, bookPath)
|
||||
defer r.Close()
|
||||
if _, err := r.TranslateBook(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Recover the manifest source the runner chunked, to compare against the --pairs column.
|
||||
manifest, err := r.bookChunks()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
bySource := map[[2]int]string{}
|
||||
for _, ch := range manifest {
|
||||
bySource[[2]int{ch.Chapter, ch.ChunkIdx}] = ch.Text
|
||||
}
|
||||
|
||||
// Default export: Source omitted (target-only contract).
|
||||
plain, err := r.Export(false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, ce := range plain.Chunks {
|
||||
if ce.Source != "" {
|
||||
t.Fatalf("default export must omit Source, got %q for ch%d/%d", ce.Source, ce.Chapter, ce.ChunkIdx)
|
||||
}
|
||||
}
|
||||
|
||||
// --pairs export: Source populated with the exact manifest source per chunk.
|
||||
pairs, err := r.Export(true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(pairs.Chunks) == 0 {
|
||||
t.Fatal("export produced no chunks")
|
||||
}
|
||||
for _, ce := range pairs.Chunks {
|
||||
want := bySource[[2]int{ce.Chapter, ce.ChunkIdx}]
|
||||
if ce.Source != want {
|
||||
t.Fatalf("--pairs ch%d/%d Source = %q, want the manifest source %q", ce.Chapter, ce.ChunkIdx, ce.Source, want)
|
||||
}
|
||||
if ce.Source == "" {
|
||||
t.Fatalf("--pairs must populate a non-empty Source for a real chunk (ch%d/%d)", ce.Chapter, ce.ChunkIdx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -95,7 +95,7 @@ func TestExportMatchesTranslateFinalText(t *testing.T) {
|
|||
|
||||
assertExportEquals := func(tag string, r *Runner) {
|
||||
t.Helper()
|
||||
exp, err := r.Export()
|
||||
exp, err := r.Export(false)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: Export: %v", tag, err)
|
||||
}
|
||||
|
|
@ -165,7 +165,7 @@ func TestExportGlossaryGateWithheld(t *testing.T) {
|
|||
if len(res.Chunks) != 1 || res.Chunks[0].FlagReason != FlagGlossaryMiss || res.Chunks[0].FinalText != "" {
|
||||
t.Fatalf("precondition: want 1 chunk flagged glossary_miss with empty text, got %+v", res.Chunks)
|
||||
}
|
||||
exp, err := r.Export()
|
||||
exp, err := r.Export(false)
|
||||
if err != nil {
|
||||
t.Fatalf("Export: %v", err)
|
||||
}
|
||||
|
|
@ -197,7 +197,7 @@ func TestExportManifestPending(t *testing.T) {
|
|||
// The ceiling stop is an infra error — expected; we only need the partial store state it leaves.
|
||||
_, _ = r.TranslateBook(ctx)
|
||||
|
||||
exp, err := r.Export()
|
||||
exp, err := r.Export(false)
|
||||
if err != nil {
|
||||
t.Fatalf("Export: %v", err)
|
||||
}
|
||||
|
|
@ -240,7 +240,7 @@ func TestExportDetectsConfigDrift(t *testing.T) {
|
|||
|
||||
// Unchanged config → no drift.
|
||||
r2 := newRunner(t, bookPath)
|
||||
exp, err := r2.Export()
|
||||
exp, err := r2.Export(false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
|
@ -263,7 +263,7 @@ func TestExportDetectsConfigDrift(t *testing.T) {
|
|||
|
||||
r3 := newRunner(t, bookPath)
|
||||
defer r3.Close()
|
||||
exp3, err := r3.Export()
|
||||
exp3, err := r3.Export(false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
|
@ -297,7 +297,7 @@ func TestExportFailLoudOnInconsistentStore(t *testing.T) {
|
|||
if err := r.Store.UpsertChunkStatus(*cs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := r.Export(); err == nil {
|
||||
if _, err := r.Export(false); err == nil {
|
||||
t.Fatal("Export must fail loud on a DispOK row with an empty final_hash")
|
||||
} else if !strings.Contains(err.Error(), "empty final_hash") {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
|
|
|
|||
|
|
@ -106,6 +106,7 @@ type memoryEntry struct {
|
|||
dst string // raw approved translation (may be "" for a ruby candidate)
|
||||
status string // auto|draft|approved
|
||||
sense string
|
||||
gender string // male|female|hidden|"" (C2) — feeds the DC3 gender constraint on the editor block
|
||||
sinceCh int
|
||||
untilCh int
|
||||
// normKeys are the normalized source surfaces (src + aliases) ELIGIBLE to fire
|
||||
|
|
@ -203,6 +204,7 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
|
|||
dst: row.Dst,
|
||||
status: row.Status,
|
||||
sense: row.Sense,
|
||||
gender: row.Gender,
|
||||
sinceCh: row.SinceCh,
|
||||
untilCh: row.UntilCh,
|
||||
allowShort: row.AllowShort,
|
||||
|
|
@ -526,11 +528,13 @@ func renderGlossaryBlock(injected []pickedEntry) string {
|
|||
|
||||
// renderFormatVersion versions the FORMAT of the role-injection renderers whose output is NOT
|
||||
// captured by memoryMatchVersion (a scope/matcher-ALGORITHM version, not a render-layout version):
|
||||
// today the editor constraint block's src→dst layout (WS2 §2в) and, later, the gender-constraint
|
||||
// render (WS5). A format change shifts the injected bytes → the wire, so it must be a loud
|
||||
// --resnapshot; folding it as a dedicated snapshot component (snapshot.go) keeps that loudness a
|
||||
// MECHANISM, not discipline — memoryMatchVersion (const memory.go) would not move on a render edit.
|
||||
const renderFormatVersion = "renderfmt-v1-editor-src2dst"
|
||||
// the editor constraint block's src→dst layout (WS2 §2в) and the DC3 gender-constraint annotation
|
||||
// (WS5 §5(б)). A format change shifts the injected bytes → the wire, so it must be a loud --resnapshot;
|
||||
// folding it as a dedicated snapshot component (snapshot.go) keeps that loudness a MECHANISM, not
|
||||
// discipline — memoryMatchVersion (const memory.go) would not move on a render edit.
|
||||
// v2 (WS5 R4): the editor constraint line now carries a gender annotation for a gendered CONFIRMED
|
||||
// term (DC3 injection — the Bai Ninbing fix: the injection DIRECTS the editor, no coreference needed).
|
||||
const renderFormatVersion = "renderfmt-v2-editor-src2dst+dc3-gender"
|
||||
|
||||
// editorConstraintHeader introduces the editor's canonical-constraint block. It gives the BILINGUAL
|
||||
// editor (D30.1) the approved src→dst bindings as consistency constraints — it must render each
|
||||
|
|
@ -570,7 +574,7 @@ func renderEditorConstraintBlock(injected []pickedEntry) string {
|
|||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
lines = append(lines, "- "+src+" → «"+dst+"»")
|
||||
lines = append(lines, "- "+src+" → «"+dst+"»"+genderConstraintNote(p.entry.gender))
|
||||
}
|
||||
if len(lines) == 0 {
|
||||
return ""
|
||||
|
|
@ -578,6 +582,23 @@ func renderEditorConstraintBlock(injected []pickedEntry) string {
|
|||
return editorConstraintHeader + "\n" + strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// genderConstraintNote is the DC3 gender directive appended to a CONFIRMED editor-constraint line (WS5
|
||||
// §5(б)): the injection DIRECTS the editor to the character's grammatical gender, which is the fix for
|
||||
// the Bai Ninbing class — no coreference needed. male/female ⇒ hard gender forms; hidden ⇒ a mandate to
|
||||
// AVOID gender-marking constructions until the reveal (a masculine default when unavoidable, D19.3). ""
|
||||
// for a term with no gender datum (the common case → the line is unchanged, byte-identical to before).
|
||||
func genderConstraintNote(gender string) string {
|
||||
switch gender {
|
||||
case "male", "m":
|
||||
return " (муж. — мужские родовые формы)"
|
||||
case "female", "f":
|
||||
return " (жен. — женские родовые формы)"
|
||||
case "hidden":
|
||||
return " (пол СКРЫТ до раскрытия — избегай родовых форм; при неизбежности — мужские)"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// --- Aho-Corasick multi-pattern automaton over runes ----------------------------
|
||||
|
||||
type acMatch struct {
|
||||
|
|
|
|||
|
|
@ -501,6 +501,58 @@ func hasAliasSurface(aliases []store.GlossaryAlias, surface string) bool {
|
|||
return false
|
||||
}
|
||||
|
||||
// SeedLint dry-runs the REAL loadGlossarySeed fail-louds + the approved shared-key collision check over
|
||||
// a glossary seed YAML (a manual seed OR the emitted mined delta), returning nil on a clean seed or a
|
||||
// loud error listing every problem (WS3 (д): "сухой прогон фейл-лаудов реального loadGlossarySeed по
|
||||
// дельте"). $0, no store, no LLM — it runs the same loader translate would, so a delta that lints clean
|
||||
// here is guaranteed loadable, and a shared-key livelock / duplicate / missing-dst is caught BEFORE the
|
||||
// W1.5 reseed instead of aborting a paid run.
|
||||
func SeedLint(path string) error {
|
||||
entries, err := loadGlossarySeed(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if problems := approvedSharedKeyCollisions(entries); len(problems) > 0 {
|
||||
return fmt.Errorf("glossary seed %s shared-key collisions:\n - %s", path, strings.Join(problems, "\n - "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// --- mined → candidates (WS3 W1.5 mined-write path) -----------------------------
|
||||
|
||||
// minedToCandidates converts the miner's WHICH proposals (MineBank) into store candidates for the W1.5
|
||||
// reseed (WS3 / §1в mined-write path). It is the DEDICATED mined path the plan requires: it stamps
|
||||
// Source:"mined" (mirroring rubyToCandidates' Source:"ruby"), NOT the Source:"seed" that loadGlossarySeed
|
||||
// hardcodes — so a mined row is EXCLUDED from the base-bank-version (BaseVersion) and the W1.5 bank
|
||||
// enrichment moves ONLY snapshot_W2, keeping W1 checkpoints valid («переоплата ОДНА», §1в/F2). Every
|
||||
// candidate is status=auto with NO dst (the WHAT is delivered by the banknote/owner sign, WS4): inert on
|
||||
// the hot path until a human/batch promotes it, never a blind name-lock. A src already covered by the
|
||||
// manual seed is skipped (the curated entry wins), exactly like rubyToCandidates. Deterministic (MineBank
|
||||
// emits a src-sorted delta; aliases are re-sorted on read by GlossaryForBook).
|
||||
func minedToCandidates(mined []MinedTerm, manualSrcs map[string]bool) []store.GlossaryEntry {
|
||||
var out []store.GlossaryEntry
|
||||
for _, m := range mined {
|
||||
if manualSrcs[m.Src] {
|
||||
continue // the curated seed already owns this src
|
||||
}
|
||||
var aliases []store.GlossaryAlias
|
||||
seen := map[string]bool{}
|
||||
for _, a := range m.Aliases {
|
||||
if a == "" || a == m.Src || seen[a] {
|
||||
continue
|
||||
}
|
||||
seen[a] = true
|
||||
aliases = append(aliases, store.GlossaryAlias{Alias: a, AliasType: "mined"})
|
||||
}
|
||||
out = append(out, store.GlossaryEntry{
|
||||
Src: m.Src, Dst: "", Type: m.Type, Status: "auto", Source: "mined",
|
||||
SinceCh: m.SinceCh, Confidence: m.Freq, Aliases: aliases,
|
||||
Note: "mined WHICH candidate (miner-v1); dst delivered by banknote/owner sign — promote before use",
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ruby classifier classes (deterministic HINTS for human promotion).
|
||||
const (
|
||||
rubyClassName = "name_candidate" // all-Han base + all-kana reading: the furigana-name SHAPE
|
||||
|
|
|
|||
152
backend/internal/pipeline/miner.go
Normal file
152
backend/internal/pipeline/miner.go
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
package pipeline
|
||||
|
||||
import "sort"
|
||||
|
||||
// miner.go: the WS3 bank-mining detector orchestration (a Go port of the frozen exp16 arm A3 = V-C,
|
||||
// LEMMATIZER-DEFAULT-B). Default B drops the pymorphy3-backed sub-channels the ratified §10-14 default
|
||||
// removes: the spread multiplier (λ=0 already zeroed it), the dst-variants co-occurrence, and the
|
||||
// Palladius ru-side name confirmation (+56 bonus). The dst (WHAT) is delivered by the banknote (WS4),
|
||||
// never by co-occurrence, so exact rank is not a product value — the invariant guarantees are the
|
||||
// candidate SET (membership), recall@proposed, and the catastrophe screen (∈top-50); dropping the
|
||||
// Palladius sub-channel moves ONLY 古月's exact rank (21→22 vs the pymorphy3 reference), screen still
|
||||
// PASS. Pure and deterministic ($0, no LLM, no network, no time/rand).
|
||||
|
||||
// MinerConfig is the frozen miner-v1 config (thresholds tuned on chapters 1–15, §D1). Values mirror
|
||||
// arms.FROZEN; the pymorphy3-dependent knobs (lam, spread_freq_min) are absent by default-B design.
|
||||
type MinerConfig struct {
|
||||
FreqFloor int
|
||||
SubsumeAlpha float64
|
||||
FormantMinPartners int
|
||||
FormantMinOverRep float64
|
||||
Bonus map[string]float64 // typ → base bonus (name/place/title/term)
|
||||
PatternWeight map[string]float64 // evidence-source → weight (max over a candidate's evidence)
|
||||
SubfloorFreqScale float64
|
||||
TopK int
|
||||
PackVersion string
|
||||
}
|
||||
|
||||
// frozenMinerConfig is the ratified default-B config (versionable — a threshold edit is a mining-config
|
||||
// bump the caller folds; a ±50% sweep is invariant, A3@f≥3=1.0 holds by floor/over_rep/subsume/partners).
|
||||
func frozenMinerConfig() MinerConfig {
|
||||
return MinerConfig{
|
||||
FreqFloor: minerFreqFloor,
|
||||
SubsumeAlpha: minerSubsumeAlpha,
|
||||
FormantMinPartners: 3,
|
||||
FormantMinOverRep: 15.0,
|
||||
Bonus: map[string]float64{"name": 140.0, "place": 140.0, "title": 120.0, "term": 80.0},
|
||||
PatternWeight: map[string]float64{
|
||||
"surname": 1.0, "ordinal_title": 1.0, "title_suffix": 0.9, "topo_suffix": 0.9,
|
||||
"rank_grade": 1.0, "formant_suffix": 0.6, "formant_prefix": 0.5, "title_bare": 0.4,
|
||||
"palladius": 0.8, // retained for parity of the weight table; the sub-channel is off in default B
|
||||
},
|
||||
SubfloorFreqScale: 40.0,
|
||||
TopK: 90,
|
||||
PackVersion: minerPackVersion,
|
||||
}
|
||||
}
|
||||
|
||||
// ScoredCand is one candidate after the V-C arm assembly (arms.ScoredCand, minus the default-B-dropped
|
||||
// dst_variants/spread). types/evidence are in deterministic first-seen order; fromPattern marks a
|
||||
// pattern-channel contribution.
|
||||
type ScoredCand struct {
|
||||
Src string
|
||||
Score float64
|
||||
Freq int
|
||||
Types []string
|
||||
Evidence []string
|
||||
FromPattern bool
|
||||
}
|
||||
|
||||
// mineResult carries the detector outputs the alias/emit layers consume.
|
||||
type mineResult struct {
|
||||
ranked []ScoredCand // A3 ranking, sorted (-score,-freq,src)
|
||||
subsumed map[string]bool // V-A ranking-dropped boundary fragments
|
||||
formants map[rune]formantInfo
|
||||
cfg MinerConfig
|
||||
}
|
||||
|
||||
// mineDetect runs V-A → V-C (arm A3, default B) over the normalized chunks and returns the ranked
|
||||
// candidate list. contrast is the general-zh reference (loaded once). Deterministic.
|
||||
func mineDetect(chunks []MinerChunk, contrast *Contrast, cfg MinerConfig) mineResult {
|
||||
va := runVA(chunks, contrast, cfg.FreqFloor, cfg.SubsumeAlpha)
|
||||
pats, formants := patternCandidates(chunks, va.candFreq, contrast, cfg.FormantMinPartners, cfg.FormantMinOverRep)
|
||||
|
||||
merged := make(map[string]*ScoredCand, len(va.ranked)+len(pats))
|
||||
for _, c := range va.ranked {
|
||||
merged[c.src] = &ScoredCand{Src: c.src, Score: c.score, Freq: c.freq} // score·(1+λ·sp) with λ=0
|
||||
}
|
||||
for cand, info := range pats {
|
||||
pw := 0.3
|
||||
for _, ev := range info.evidence {
|
||||
if w, ok := cfg.PatternWeight[evSource(ev)]; ok && w > pw {
|
||||
pw = w
|
||||
}
|
||||
}
|
||||
typ := "term"
|
||||
if len(info.types) > 0 {
|
||||
typ = info.types[0]
|
||||
}
|
||||
base, ok := cfg.Bonus[typ]
|
||||
if !ok {
|
||||
base = cfg.Bonus["term"]
|
||||
}
|
||||
bonus := base * pw
|
||||
if sc, ok := merged[cand]; ok {
|
||||
sc.Score += bonus
|
||||
for _, t := range info.types {
|
||||
if !containsStr(sc.Types, t) {
|
||||
sc.Types = append(sc.Types, t)
|
||||
}
|
||||
}
|
||||
sc.Evidence = append(sc.Evidence, firstN(info.evidence, 3)...)
|
||||
sc.FromPattern = true
|
||||
} else {
|
||||
f := countOccurrences(cand, chunks)
|
||||
merged[cand] = &ScoredCand{
|
||||
Src: cand, Score: bonus + cfg.SubfloorFreqScale*float64(f)*pw, Freq: f,
|
||||
Types: append([]string(nil), info.types...), Evidence: firstN(info.evidence, 3),
|
||||
FromPattern: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
// Palladius ru-side confirmation loop is INTENTIONALLY OMITTED (default B — it needs dst_variants +
|
||||
// is_name_lemma, both pymorphy3; it only shifts 古月's exact rank, which is not a product value).
|
||||
|
||||
ranked := make([]ScoredCand, 0, len(merged))
|
||||
for _, sc := range merged {
|
||||
ranked = append(ranked, *sc)
|
||||
}
|
||||
sortScored(ranked)
|
||||
return mineResult{ranked: ranked, subsumed: va.subsumed, formants: formants, cfg: cfg}
|
||||
}
|
||||
|
||||
// sortScored orders the arm ranking by (-score, -freq, src) — the frozen key.
|
||||
func sortScored(cs []ScoredCand) {
|
||||
sort.Slice(cs, func(i, j int) bool {
|
||||
if cs[i].Score != cs[j].Score {
|
||||
return cs[i].Score > cs[j].Score
|
||||
}
|
||||
if cs[i].Freq != cs[j].Freq {
|
||||
return cs[i].Freq > cs[j].Freq
|
||||
}
|
||||
return cs[i].Src < cs[j].Src
|
||||
})
|
||||
}
|
||||
|
||||
// evSource is the pattern-weight key: the evidence tag before its ':' (arms' ev.split(":")[0]).
|
||||
func evSource(ev string) string {
|
||||
for i := 0; i < len(ev); i++ {
|
||||
if ev[i] == ':' {
|
||||
return ev[:i]
|
||||
}
|
||||
}
|
||||
return ev
|
||||
}
|
||||
|
||||
// firstN returns up to n elements of ss (a fresh slice, never aliasing the input).
|
||||
func firstN(ss []string, n int) []string {
|
||||
if len(ss) < n {
|
||||
n = len(ss)
|
||||
}
|
||||
return append([]string(nil), ss[:n]...)
|
||||
}
|
||||
194
backend/internal/pipeline/miner_alias.go
Normal file
194
backend/internal/pipeline/miner_alias.go
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// miner_alias.go: alias tier-1 clustering (WS3 — a Go port of exp16 alias.py, DEFAULT-B). Precision-safe
|
||||
// PROP-layer rules over a set of src surfaces:
|
||||
//
|
||||
// R1 surface containment X ⊂ Y, |X|≥2 → 'extension' edge (方源 ⊂ 古月方源), COMPOSITIONAL guard
|
||||
// (古月+族长 = a phrase, not an alias) blocks the clan↔title chaining
|
||||
// R2 surname anchor+compose shared surname → FAMILY supercluster (NOT identity)
|
||||
// R4 NEGATIVE hard-blocks (checked FIRST): (ii) different confirmed gender; (iii/v) different approved
|
||||
// dst → different entities EVEN under containment (族长⊂四代族长, 古月⊂古月方源); (i) same surname +
|
||||
// different given names → family, not identity; (iv) co-presence in one source sentence → not identity
|
||||
//
|
||||
// The R3 "shared ru rendering → identity" edge is OMITTED in default B (it needs the pymorphy3-backed
|
||||
// dst-variant lemmas, which the ratified default drops). Precision is measured entity-level (B³, §A5).
|
||||
|
||||
// aliasParticle is the trailing-particle set that marks a boundary FRAGMENT (alias.PARTICLE): a candidate
|
||||
// = a known surface + one of these is 方源在/方源心, not an entity.
|
||||
var aliasParticle = buildRuneSet2("的了在是和就也都不心面前后上下今少多大小与之其")
|
||||
|
||||
// aliasSurface is one src surface with the seed metadata the rules read (alias.Surface, minus the
|
||||
// default-B-dropped dst_lemmas).
|
||||
type aliasSurface struct {
|
||||
src string // normalized src surface
|
||||
gender string
|
||||
approvedDst string
|
||||
typ string
|
||||
}
|
||||
|
||||
// aliasEdge is one proposed edge between two surfaces.
|
||||
type aliasEdge struct {
|
||||
a, b string
|
||||
rule string
|
||||
kind string
|
||||
}
|
||||
|
||||
// isFragment reports whether src is a known surface + a trailing particle (alias.frag): a boundary
|
||||
// artifact, not an independent entity. seedSurfaces is the normalized seed src/alias set.
|
||||
func isFragment(src string, seedSurfaces map[string]bool) bool {
|
||||
r := []rune(src)
|
||||
if len(r) < 2 {
|
||||
return false
|
||||
}
|
||||
return seedSurfaces[string(r[:len(r)-1])] && aliasParticle[r[len(r)-1]]
|
||||
}
|
||||
|
||||
// cooccurSameSentence reports whether a and b appear in one source sentence anywhere (alias.
|
||||
// cooccur_same_sentence: split on 。!?\n).
|
||||
func cooccurSameSentence(chunks []MinerChunk, aNorm, bNorm string) bool {
|
||||
for _, c := range chunks {
|
||||
for _, sent := range splitMinerSentences(c.NSource) {
|
||||
if strings.Contains(sent, aNorm) && strings.Contains(sent, bNorm) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func splitMinerSentences(s string) []string {
|
||||
return strings.FieldsFunc(s, func(r rune) bool {
|
||||
return r == '。' || r == '!' || r == '?' || r == '\n'
|
||||
})
|
||||
}
|
||||
|
||||
// proposeAliasEdges applies the tier-1 rules over the surfaces, returning identity, family and weak
|
||||
// edges (alias.propose_edges, default B). seedSurfaces is the normalized seed src/alias set (the R1
|
||||
// entity-vs-fragment guard). Deterministic: surfaces are iterated in a fixed sorted order.
|
||||
func proposeAliasEdges(surfaces map[string]aliasSurface, chunks []MinerChunk, seedSurfaces map[string]bool) (ident, family, weak []aliasEdge) {
|
||||
keys := make([]string, 0, len(surfaces))
|
||||
for k := range surfaces {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for i := 0; i < len(keys); i++ {
|
||||
for j := i + 1; j < len(keys); j++ {
|
||||
a, b := surfaces[keys[i]], surfaces[keys[j]]
|
||||
// R4-ii different confirmed gender (hidden never blocks).
|
||||
if a.gender != "" && b.gender != "" && a.gender != b.gender && a.gender != "hidden" && b.gender != "hidden" {
|
||||
continue
|
||||
}
|
||||
// R4-iii/v different approved dst → different entities, even under containment.
|
||||
if a.approvedDst != "" && b.approvedDst != "" && normalizeSourceKey(a.approvedDst) != normalizeSourceKey(b.approvedDst) {
|
||||
continue
|
||||
}
|
||||
surA, surB := isSurnameStart([]rune(a.src)), isSurnameStart([]rune(b.src))
|
||||
// R1 containment → extension edge (same entity, fuller vs shorter surface).
|
||||
if runeLen(a.src) >= 2 && runeLen(b.src) >= 2 && a.src != b.src && (strings.Contains(b.src, a.src) || strings.Contains(a.src, b.src)) {
|
||||
short, long := a, b
|
||||
if runeLen(b.src) < runeLen(a.src) {
|
||||
short, long = b, a
|
||||
}
|
||||
shortIsEntity := short.typ != "" || seedSurfaces[short.src]
|
||||
rest := containmentRest(short.src, long.src)
|
||||
restEnt, restKnown := surfaces[rest]
|
||||
compositional := rest != "" && seedSurfaces[rest] &&
|
||||
(!restKnown || restEnt.typ == "title" || restEnt.typ == "place" || restEnt.typ == "term")
|
||||
switch {
|
||||
case compositional:
|
||||
weak = append(weak, aliasEdge{a.src, b.src, "R1-compositional", "phrase_not_alias"})
|
||||
case shortIsEntity:
|
||||
ident = append(ident, aliasEdge{a.src, b.src, "R1", "extension"})
|
||||
default:
|
||||
weak = append(weak, aliasEdge{a.src, b.src, "R1-fragment?", "containment_weak"})
|
||||
}
|
||||
continue
|
||||
}
|
||||
// R2 surname anchor → family (NOT identity).
|
||||
if surA != "" && surB != "" && surA == surB && a.src != b.src {
|
||||
givenA := trimRunePrefix(a.src, surA)
|
||||
givenB := trimRunePrefix(b.src, surB)
|
||||
if givenA != givenB && givenA != "" && givenB != "" { // R4-i different given names → family only
|
||||
if cooccurSameSentence(chunks, a.src, b.src) {
|
||||
family = append(family, aliasEdge{a.src, b.src, "R2+R4iv", "family_copresent"})
|
||||
} else {
|
||||
family = append(family, aliasEdge{a.src, b.src, "R2", "family"})
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
// R3 (shared ru rendering → identity) is OMITTED in default B.
|
||||
}
|
||||
}
|
||||
return ident, family, weak
|
||||
}
|
||||
|
||||
// containmentRest is alias.py's rest = long[len(short):] if long.startswith(short) else long[:-len(short)]
|
||||
// (by character). short is contained in long.
|
||||
func containmentRest(short, long string) string {
|
||||
sr, lr := []rune(short), []rune(long)
|
||||
if len(sr) > len(lr) {
|
||||
return ""
|
||||
}
|
||||
if runesEqual(lr[:len(sr)], sr) {
|
||||
return string(lr[len(sr):])
|
||||
}
|
||||
return string(lr[:len(lr)-len(sr)])
|
||||
}
|
||||
|
||||
// trimRunePrefix removes the leading prefix (a surname) from src, by character.
|
||||
func trimRunePrefix(src, prefix string) string {
|
||||
sr, pr := []rune(src), []rune(prefix)
|
||||
if len(pr) <= len(sr) && runesEqual(sr[:len(pr)], pr) {
|
||||
return string(sr[len(pr):])
|
||||
}
|
||||
return src
|
||||
}
|
||||
|
||||
// clusterAlias runs union-find over the identity edges and returns clusters of size > 1, each sorted
|
||||
// (alias.cluster). allSurfaces bounds the universe.
|
||||
func clusterAlias(ident []aliasEdge, allSurfaces []string) [][]string {
|
||||
parent := make(map[string]string, len(allSurfaces))
|
||||
for _, s := range allSurfaces {
|
||||
parent[s] = s
|
||||
}
|
||||
var find func(string) string
|
||||
find = func(x string) string {
|
||||
for parent[x] != x {
|
||||
parent[x] = parent[parent[x]]
|
||||
x = parent[x]
|
||||
}
|
||||
return x
|
||||
}
|
||||
for _, e := range ident {
|
||||
if _, ok := parent[e.a]; ok {
|
||||
if _, ok := parent[e.b]; ok {
|
||||
parent[find(e.a)] = find(e.b)
|
||||
}
|
||||
}
|
||||
}
|
||||
groups := map[string][]string{}
|
||||
for _, s := range allSurfaces {
|
||||
r := find(s)
|
||||
groups[r] = append(groups[r], s)
|
||||
}
|
||||
var out [][]string
|
||||
roots := make([]string, 0, len(groups))
|
||||
for r := range groups {
|
||||
roots = append(roots, r)
|
||||
}
|
||||
sort.Strings(roots)
|
||||
for _, r := range roots {
|
||||
if len(groups[r]) > 1 {
|
||||
g := append([]string(nil), groups[r]...)
|
||||
sort.Strings(g)
|
||||
out = append(out, g)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
160
backend/internal/pipeline/miner_detect.go
Normal file
160
backend/internal/pipeline/miner_detect.go
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"math"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// miner_detect.go: the deterministic V-A detector (WS3 — a Go port of exp16 detectors.py). Frequency ×
|
||||
// contrast(weirdness) × c-value(nested discount). Output: ranked src candidates. The thresholds (freq
|
||||
// floor 3, subsume α 0.80) are frozen miner-v1 (tuned on chapters 1–15, §D1). Pure and deterministic —
|
||||
// map iteration only ever precedes a total sort by (-score, -freq, src).
|
||||
|
||||
// minerSubsumeAlpha: drop candidate a if a longer container b has f(b) >= α·f(a) — a is a boundary
|
||||
// fragment, not an independent term (detectors.SUBSUME_ALPHA). It only cleans the candidate RANKING;
|
||||
// the occurrence axis (countOccurrences) is suppress-free (D24.2).
|
||||
const minerSubsumeAlpha = 0.80
|
||||
|
||||
// runeLen is the character length of a candidate (Han n-grams are ≤6 runes).
|
||||
func runeLen(s string) int { return utf8.RuneCountInString(s) }
|
||||
|
||||
// minerCandidate is one V-A-scored candidate (detectors.Candidate; the V-B/V-C enrichments live on the
|
||||
// arm-level ScoredCand, not here).
|
||||
type minerCandidate struct {
|
||||
src string
|
||||
freq int
|
||||
cvalue float64
|
||||
weirdness float64
|
||||
termhood float64
|
||||
score float64
|
||||
length int
|
||||
}
|
||||
|
||||
// computeCValue is the c-value with nested discount (detectors.compute_cvalue). For a candidate a nested
|
||||
// in longer candidates T_a: cval(a) = g(|a|)·(f(a) − (1/|T_a|)·Σ_{b∈T_a} f(b)); non-nested: g(|a|)·f(a).
|
||||
// g(L)=log2(L+1). Deterministic (accumulates into maps, order-independent).
|
||||
func computeCValue(candFreq map[string]int) map[string]float64 {
|
||||
tcount := make(map[string]int, len(candFreq))
|
||||
tfreqsum := make(map[string]int, len(candFreq))
|
||||
for b, fb := range candFreq {
|
||||
br := []rune(b)
|
||||
Lb := len(br)
|
||||
if Lb < 2 {
|
||||
continue
|
||||
}
|
||||
// Distinct proper substrings a of b that are themselves candidates.
|
||||
subs := map[string]bool{}
|
||||
for n := 1; n < Lb; n++ {
|
||||
for i := 0; i+n <= Lb; i++ {
|
||||
a := string(br[i : i+n])
|
||||
if a == b {
|
||||
continue
|
||||
}
|
||||
if _, ok := candFreq[a]; ok {
|
||||
subs[a] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
for a := range subs {
|
||||
tcount[a]++
|
||||
tfreqsum[a] += fb
|
||||
}
|
||||
}
|
||||
cval := make(map[string]float64, len(candFreq))
|
||||
for a, fa := range candFreq {
|
||||
g := lengthMult(runeLen(a))
|
||||
if tcount[a] > 0 {
|
||||
cval[a] = g * (float64(fa) - float64(tfreqsum[a])/float64(tcount[a]))
|
||||
} else {
|
||||
cval[a] = g * float64(fa)
|
||||
}
|
||||
}
|
||||
return cval
|
||||
}
|
||||
|
||||
// subsumedCandidates returns the boundary-fragment candidates to drop from the RANKING (detectors.
|
||||
// subsumed_candidates): a whose every occurrence is (almost) inside a strictly longer candidate b with
|
||||
// f(b) ≥ α·f(a). 方源(559) survives 方源的(85) [ratio 0.15]; 花酒行(65) is dropped by 花酒行者(65) [1.0].
|
||||
func subsumedCandidates(candFreq map[string]int, alpha float64) map[string]bool {
|
||||
byLen := map[int][]string{}
|
||||
for c := range candFreq {
|
||||
L := runeLen(c)
|
||||
byLen[L] = append(byLen[L], c)
|
||||
}
|
||||
drop := map[string]bool{}
|
||||
for a, fa := range candFreq {
|
||||
La := runeLen(a)
|
||||
found := false
|
||||
for Lb := La + 1; Lb <= minerNgramMax && !found; Lb++ {
|
||||
for _, b := range byLen[Lb] {
|
||||
if strings.Contains(b, a) && float64(candFreq[b]) >= alpha*float64(fa) {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if found {
|
||||
drop[a] = true
|
||||
}
|
||||
}
|
||||
return drop
|
||||
}
|
||||
|
||||
// vaResult carries the V-A build outputs the arms/patterns layers need.
|
||||
type vaResult struct {
|
||||
candFreq map[string]int // freq ≥ floor
|
||||
nBook int // total n-gram token mass (weirdness denominator scale)
|
||||
subsumed map[string]bool // ranking-dropped boundary fragments
|
||||
cval map[string]float64 // per-candidate c-value
|
||||
ranked []minerCandidate // V-A ranking (subsumed excluded), sorted (-score,-freq,src)
|
||||
}
|
||||
|
||||
// runVA builds and scores V-A over the chunks (detectors.VA.build + score_all). candFreq is the
|
||||
// freq≥floor multiset; nBook is the FULL (pre-floor) token mass. score = max(cval,0)·termhood, where
|
||||
// termhood=log1p(weirdness), weirdness = P_book / max(word_rel, char_indep_rel, 1e-9).
|
||||
func runVA(chunks []MinerChunk, contrast *Contrast, freqFloor int, alpha float64) vaResult {
|
||||
allFreq, nBook := enumerateCandidates(chunks)
|
||||
candFreq := make(map[string]int, len(allFreq))
|
||||
for k, v := range allFreq {
|
||||
if v >= freqFloor {
|
||||
candFreq[k] = v
|
||||
}
|
||||
}
|
||||
subsumed := subsumedCandidates(candFreq, alpha)
|
||||
cval := computeCValue(candFreq)
|
||||
|
||||
const weirdFloor = 1e-9
|
||||
var ranked []minerCandidate
|
||||
for a, f := range candFreq {
|
||||
if subsumed[a] {
|
||||
continue
|
||||
}
|
||||
pBook := float64(f) / float64(nBook)
|
||||
gen := math.Max(math.Max(contrast.wordRel(a), contrast.charIndepRel(a)), weirdFloor)
|
||||
w := pBook / gen
|
||||
termhood := math.Log1p(w)
|
||||
cv := cval[a]
|
||||
score := math.Max(cv, 0.0) * termhood
|
||||
ranked = append(ranked, minerCandidate{
|
||||
src: a, freq: f, cvalue: cv, weirdness: w, termhood: termhood, score: score, length: runeLen(a),
|
||||
})
|
||||
}
|
||||
sortCandidates(ranked)
|
||||
return vaResult{candFreq: candFreq, nBook: nBook, subsumed: subsumed, cval: cval, ranked: ranked}
|
||||
}
|
||||
|
||||
// sortCandidates orders by (-score, -freq, src) — the frozen miner-v1 ranking key (stable across Go/
|
||||
// Python via the same tie-break).
|
||||
func sortCandidates(cs []minerCandidate) {
|
||||
sort.Slice(cs, func(i, j int) bool {
|
||||
if cs[i].score != cs[j].score {
|
||||
return cs[i].score > cs[j].score
|
||||
}
|
||||
if cs[i].freq != cs[j].freq {
|
||||
return cs[i].freq > cs[j].freq
|
||||
}
|
||||
return cs[i].src < cs[j].src
|
||||
})
|
||||
}
|
||||
227
backend/internal/pipeline/miner_emit.go
Normal file
227
backend/internal/pipeline/miner_emit.go
Normal file
|
|
@ -0,0 +1,227 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
|
||||
"textmachine/backend/internal/store"
|
||||
)
|
||||
|
||||
// miner_emit.go: the WS3 seed-delta emission (§C2-7) — the miner's owner-facing output. It runs the
|
||||
// default-B detector, applies the subsumption + fragment + type + freq FILTERS (never a raw 13618-dump,
|
||||
// which carries §B2 fragment noise), clusters aliases (tier-1), and emits one MinedTerm per NEW entity.
|
||||
//
|
||||
// Discipline: the miner NEVER writes `approved` — every emitted term is a `draft`/`auto` PROPOSAL for the
|
||||
// owner to sign (§C2). In default B the WHAT (dst) is delivered by the banknote (WS4) at the W1.5 sign
|
||||
// boundary, so the miner emits WHICH-only candidates (status:auto, no dst — inert until a dst + owner
|
||||
// promotion). A candidate that clusters with an existing seed surface is an alias-of-existing and is left
|
||||
// for the owner to attach (not emitted as a new entity), so the delta is genuinely new terms.
|
||||
|
||||
// emitMinFreq is the emission frequency floor (emit_owner_sheets.build_precision30: freq ≥ 5).
|
||||
const emitMinFreq = 5
|
||||
|
||||
// MinedTerm is one WHICH candidate the miner proposes (a seed-delta entry). The miner never writes a dst
|
||||
// (default B) or `approved`; the caller persists it via the mined-write path (Source:"mined", status
|
||||
// auto) and the owner signs it at W1.5.
|
||||
type MinedTerm struct {
|
||||
Src string
|
||||
Type string // name|place|title|term (the candidate's first pattern type; "" → term)
|
||||
SinceCh int // first chapter of appearance across the term + its aliases (auto)
|
||||
Freq int
|
||||
Aliases []string // identity-cluster co-surfaces (normalized), sorted; the entity's other surfaces
|
||||
Evidence []string // up to 3 pattern-evidence tags (for the owner sidecar)
|
||||
}
|
||||
|
||||
// MineBank runs the full default-B miner over the normalized chunks and emits the alias-clustered,
|
||||
// filtered mined delta (WHICH candidates for owner sign). seed is the current glossary (its surfaces
|
||||
// scope the non-seed filter, the fragment guard, and the alias entity/metadata). Deterministic and $0.
|
||||
func MineBank(chunks []MinerChunk, contrast *Contrast, seed []store.GlossaryEntry, cfg MinerConfig) []MinedTerm {
|
||||
mr := mineDetect(chunks, contrast, cfg)
|
||||
|
||||
// Seed surfaces (normalized) + their metadata for the alias rules and the non-seed / fragment guards.
|
||||
seedSurfaces := map[string]bool{}
|
||||
seedMeta := map[string]aliasSurface{}
|
||||
for _, e := range seed {
|
||||
for _, surf := range append([]string{e.Src}, aliasStrings(e.Aliases)...) {
|
||||
nk := normalizeSourceKey(surf)
|
||||
if nk == "" {
|
||||
continue
|
||||
}
|
||||
seedSurfaces[nk] = true
|
||||
// The primary src carries the dst/gender/type; an alias inherits them (same entity).
|
||||
if _, ok := seedMeta[nk]; !ok {
|
||||
meta := aliasSurface{src: nk, typ: e.Type}
|
||||
if e.Status == "approved" {
|
||||
meta.approvedDst = e.Dst
|
||||
}
|
||||
meta.gender = e.Gender
|
||||
seedMeta[nk] = meta
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Qualifying candidate pool (build_precision30 filters), in ranked order (the ranked order is the
|
||||
// representative-selection order below — the top-ranked cluster member owns the aliases).
|
||||
var pool []ScoredCand
|
||||
for _, c := range mr.ranked {
|
||||
if !emissionEligible(c, mr.subsumed, seedSurfaces) {
|
||||
continue
|
||||
}
|
||||
pool = append(pool, c)
|
||||
}
|
||||
|
||||
// Alias universe = qualifying candidate surfaces ∪ seed surfaces; run tier-1 clustering.
|
||||
surfaces := map[string]aliasSurface{}
|
||||
for _, c := range pool {
|
||||
typ := ""
|
||||
if len(c.Types) > 0 {
|
||||
typ = c.Types[0]
|
||||
}
|
||||
surfaces[c.Src] = aliasSurface{src: c.Src, typ: typ}
|
||||
}
|
||||
for nk, meta := range seedMeta {
|
||||
surfaces[nk] = meta // seed metadata wins (gender/approved_dst)
|
||||
}
|
||||
universe := make([]string, 0, len(surfaces))
|
||||
for s := range surfaces {
|
||||
universe = append(universe, s)
|
||||
}
|
||||
sort.Strings(universe)
|
||||
ident, _, _ := proposeAliasEdges(surfaces, chunks, seedSurfaces)
|
||||
clusters := clusterAlias(ident, universe)
|
||||
|
||||
// clusterOf maps a surface to its identity cluster (members); a surface not in a multi-cluster maps
|
||||
// to a singleton.
|
||||
clusterOf := map[string][]string{}
|
||||
for _, cl := range clusters {
|
||||
for _, s := range cl {
|
||||
clusterOf[s] = cl
|
||||
}
|
||||
}
|
||||
|
||||
var out []MinedTerm
|
||||
emitted := map[string]bool{}
|
||||
for _, c := range pool { // ranked order → the top-ranked cluster member is the representative
|
||||
if seedSurfaces[c.Src] || emitted[c.Src] {
|
||||
continue
|
||||
}
|
||||
cl := clusterOf[c.Src]
|
||||
// A cluster touching a seed surface is an alias-of-existing entity → the owner attaches it; skip.
|
||||
if clusterHasSeed(cl, seedSurfaces) {
|
||||
markEmitted(cl, emitted)
|
||||
continue
|
||||
}
|
||||
var aliases []string
|
||||
if len(cl) > 1 {
|
||||
for _, m := range cl {
|
||||
if m != c.Src && !seedSurfaces[m] {
|
||||
aliases = append(aliases, m)
|
||||
}
|
||||
}
|
||||
sort.Strings(aliases)
|
||||
markEmitted(cl, emitted) // the whole cluster is represented by this one term
|
||||
} else {
|
||||
emitted[c.Src] = true
|
||||
}
|
||||
typ := "term"
|
||||
if len(c.Types) > 0 {
|
||||
typ = c.Types[0]
|
||||
}
|
||||
out = append(out, MinedTerm{
|
||||
Src: c.Src, Type: typ, Freq: c.Freq, Aliases: aliases, Evidence: c.Evidence,
|
||||
SinceCh: entitySinceCh(append([]string{c.Src}, aliases...), chunks),
|
||||
})
|
||||
}
|
||||
// Deterministic output order: by src (the caller may re-sort, but pin a stable delta).
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Src < out[j].Src })
|
||||
return out
|
||||
}
|
||||
|
||||
// MinedDeltaYAML serializes the mined delta into the seedTerm YAML schema (§C2-7) — the owner-sign
|
||||
// artifact and the `tmctl seed-lint` input. Every term is status:auto with NO dst (WHICH-only, default
|
||||
// B; the WHAT is delivered by the banknote at W1.5). It reuses the EXISTING seedTerm schema (no new
|
||||
// fields — §3(б)); evidence / zones live in the sidecar sign-map, not the seed. The output is guaranteed
|
||||
// loadable by loadGlossarySeed (status:auto may lack a dst); seed-lint proves it against the real loader.
|
||||
func MinedDeltaYAML(mined []MinedTerm) (string, error) {
|
||||
sf := seedFile{}
|
||||
for _, m := range mined {
|
||||
st := seedTerm{Src: m.Src, Type: m.Type, Status: "auto", SinceCh: m.SinceCh}
|
||||
for _, a := range m.Aliases {
|
||||
st.Aliases = append(st.Aliases, seedAlias{Alias: a, Type: "mined"})
|
||||
}
|
||||
if len(m.Evidence) > 0 {
|
||||
st.Note = "mined WHICH candidate; evidence: " + fmt.Sprint(m.Evidence)
|
||||
}
|
||||
sf.Terms = append(sf.Terms, st)
|
||||
}
|
||||
b, err := yaml.Marshal(sf)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("pipeline: marshal mined delta: %w", err)
|
||||
}
|
||||
return string(b), nil
|
||||
}
|
||||
|
||||
// emissionEligible applies the build_precision30 filters: type ∈ {name,place,title}, freq ≥ 5, src not
|
||||
// subsumed, len ≥ 2 rune, not a boundary fragment.
|
||||
func emissionEligible(c ScoredCand, subsumed, seedSurfaces map[string]bool) bool {
|
||||
if !hasAnyType(c.Types, "name", "place", "title") {
|
||||
return false
|
||||
}
|
||||
if c.Freq < emitMinFreq || subsumed[c.Src] || runeLen(c.Src) < 2 {
|
||||
return false
|
||||
}
|
||||
if isFragment(c.Src, seedSurfaces) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func hasAnyType(types []string, want ...string) bool {
|
||||
for _, t := range types {
|
||||
for _, w := range want {
|
||||
if t == w {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func clusterHasSeed(cl []string, seedSurfaces map[string]bool) bool {
|
||||
for _, s := range cl {
|
||||
if seedSurfaces[s] {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func markEmitted(cl []string, emitted map[string]bool) {
|
||||
for _, s := range cl {
|
||||
emitted[s] = true
|
||||
}
|
||||
}
|
||||
|
||||
// entitySinceCh is the earliest chapter any of the entity's surfaces appears in (canon.since_ch = first
|
||||
// chapter of appearance).
|
||||
func entitySinceCh(surfaces []string, chunks []MinerChunk) int {
|
||||
min := 0
|
||||
for _, s := range surfaces {
|
||||
for ch := range candidateChapters(s, chunks) {
|
||||
if min == 0 || ch < min {
|
||||
min = ch
|
||||
}
|
||||
}
|
||||
}
|
||||
return min
|
||||
}
|
||||
|
||||
func aliasStrings(as []store.GlossaryAlias) []string {
|
||||
out := make([]string, len(as))
|
||||
for i, a := range as {
|
||||
out[i] = a.Alias
|
||||
}
|
||||
return out
|
||||
}
|
||||
138
backend/internal/pipeline/miner_palladius.go
Normal file
138
backend/internal/pipeline/miner_palladius.go
Normal file
|
|
@ -0,0 +1,138 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// miner_palladius.go: the Palladius (Палладий) transliteration syllable table + ru-side name-shape
|
||||
// detector (WS3 — a Go port of exp16 palladius.py). A ru token that segments cleanly into Palladius
|
||||
// syllables looks like a transliterated Chinese NAME. The pinyin→Cyrillic table is generated
|
||||
// deterministically from initials × finals + the documented irregularities (a versioned artifact).
|
||||
//
|
||||
// In the ratified DEFAULT-B miner the ru-side name CONFIRMATION sub-channel is OFF (it also needs the
|
||||
// pymorphy3 is_name_lemma gate to block the «найти»=най+ти false-positive class), so isPalladiusToken is
|
||||
// not wired into the default-B detector pipeline. It is kept as a deterministic, tested artifact — the
|
||||
// name-shape signal a future faithful-A variant (with a Go morphology backend) would consume, and the
|
||||
// pattern P4 transliteration-by-pair seam (§B5).
|
||||
|
||||
// palladius initials (声母) → Palladius Cyrillic (palladius.INITIALS).
|
||||
var palladiusInitials = map[string]string{
|
||||
"b": "б", "p": "п", "m": "м", "f": "ф", "d": "д", "t": "т", "n": "н", "l": "л",
|
||||
"g": "г", "k": "к", "h": "х", "j": "цз", "q": "ц", "x": "с",
|
||||
"zh": "чж", "ch": "ч", "sh": "ш", "r": "ж", "z": "цз", "c": "ц", "s": "с",
|
||||
}
|
||||
|
||||
// palladius finals (韵母) → Palladius Cyrillic (palladius.FINALS). ü is written as v.
|
||||
var palladiusFinals = map[string]string{
|
||||
"a": "а", "o": "о", "e": "э", "ai": "ай", "ei": "эй", "ao": "ао", "ou": "оу",
|
||||
"an": "ань", "en": "энь", "ang": "ан", "eng": "эн", "er": "эр", "ong": "ун",
|
||||
"i": "и", "ia": "я", "ie": "е", "iao": "яо", "iu": "ю", "ian": "янь",
|
||||
"in": "инь", "iang": "ян", "ing": "ин", "iong": "юн",
|
||||
"u": "у", "ua": "уа", "uo": "о", "uai": "уай", "ui": "уй", "uan": "уань",
|
||||
"un": "унь", "uang": "уан", "ueng": "ун",
|
||||
"v": "юй", "ve": "юэ", "van": "юань", "vn": "юнь",
|
||||
}
|
||||
|
||||
// whole zero-initial (y-/w-) syllables (palladius.Y_W).
|
||||
var palladiusYW = map[string]string{
|
||||
"yi": "и", "ya": "я", "ye": "е", "yao": "яо", "you": "ю", "yan": "янь", "yin": "инь",
|
||||
"yang": "ян", "ying": "ин", "yong": "юн", "yu": "юй", "yue": "юэ", "yuan": "юань", "yun": "юнь",
|
||||
"wu": "у", "wa": "ва", "wo": "во", "wai": "вай", "wei": "вэй", "wan": "вань", "wen": "вэнь",
|
||||
"wang": "ван", "weng": "вэн",
|
||||
}
|
||||
|
||||
// retroflex/sibilant + i → -ы/-и class (palladius.SPECIAL_I).
|
||||
var palladiusSpecialI = map[string]string{
|
||||
"zhi": "чжи", "chi": "чи", "shi": "ши", "ri": "жи", "zi": "цзы", "ci": "цы", "si": "сы",
|
||||
}
|
||||
|
||||
var palladiusJQX = map[string]bool{"j": true, "q": true, "x": true}
|
||||
|
||||
// palladiusCyrSyllables is the distinct Cyrillic syllable set, longest-first, for greedy segmentation
|
||||
// (palladius._CYR_SYL). Ties in length never both match a position (distinct strings), so the order
|
||||
// among equal-length forms is immaterial — sorted (len desc, then value) for a stable artifact.
|
||||
var palladiusCyrSyllables = buildPalladiusCyrSyllables()
|
||||
|
||||
func buildPalladiusCyrSyllables() []string {
|
||||
set := map[string]bool{}
|
||||
for _, v := range palladiusYW {
|
||||
set[v] = true
|
||||
}
|
||||
for _, v := range palladiusSpecialI {
|
||||
set[v] = true
|
||||
}
|
||||
retroflex := map[string]bool{"zh": true, "ch": true, "sh": true, "r": true, "z": true, "c": true, "s": true}
|
||||
for pi, ci := range palladiusInitials {
|
||||
for pf, cf := range palladiusFinals {
|
||||
// ü finals (written v) are valid only after j/q/x (pinyin writes ü as plain u) or l/n.
|
||||
switch pf {
|
||||
case "v", "ve", "van", "vn":
|
||||
if !palladiusJQX[pi] && pi != "l" && pi != "n" {
|
||||
continue
|
||||
}
|
||||
}
|
||||
// skip the retroflex/sibilant + bare i (handled by SPECIAL_I).
|
||||
if pf == "i" && retroflex[pi] {
|
||||
continue
|
||||
}
|
||||
set[ci+cf] = true
|
||||
}
|
||||
}
|
||||
out := make([]string, 0, len(set))
|
||||
for s := range set {
|
||||
if s != "" {
|
||||
out = append(out, s)
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if len([]rune(out[i])) != len([]rune(out[j])) {
|
||||
return len([]rune(out[i])) > len([]rune(out[j]))
|
||||
}
|
||||
return out[i] < out[j]
|
||||
})
|
||||
return out
|
||||
}
|
||||
|
||||
// isPalladiusToken reports whether a ru word segments fully into Palladius syllables (greedy longest-
|
||||
// match, palladius.is_palladius_token): a high-precision "this looks like a transliterated Chinese name"
|
||||
// signal. ъ is dropped before segmentation. minSyllables is the minimum syllable count.
|
||||
func isPalladiusToken(token string, minSyllables int) bool {
|
||||
t := strings.ToLower(strings.TrimSpace(token))
|
||||
t = strings.ReplaceAll(t, "ъ", "")
|
||||
if t == "" || !allCyrillic(t) {
|
||||
return false
|
||||
}
|
||||
tr := []rune(t)
|
||||
nSyl := 0
|
||||
for i := 0; i < len(tr); {
|
||||
matched := false
|
||||
for _, s := range palladiusCyrSyllables {
|
||||
sr := []rune(s)
|
||||
if i+len(sr) <= len(tr) && runesEqual(tr[i:i+len(sr)], sr) {
|
||||
i += len(sr)
|
||||
nSyl++
|
||||
matched = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !matched {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return nSyl >= minSyllables
|
||||
}
|
||||
|
||||
// allCyrillic reports whether every rune of s is Cyrillic (palladius._RE_CYR_WORD).
|
||||
func allCyrillic(s string) bool {
|
||||
if s == "" {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if !unicode.Is(unicode.Cyrillic, r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
121
backend/internal/pipeline/miner_parity_test.go
Normal file
121
backend/internal/pipeline/miner_parity_test.go
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// miner_parity_test.go: the WS3 (д) Go↔Python full-book parity check. It re-runs the default-B miner on
|
||||
// the PINNED exp16 inputs (jieba contrast + the 25-chapter records.json + the seed GT) and asserts the
|
||||
// Palladius-INVARIANT guarantees the Go port must reproduce (research/20 §D, ws3_miner_verify.py):
|
||||
//
|
||||
// - candidate SET count = 13618 (membership, integer-determined — Palladius-invariant);
|
||||
// - recall@proposed (overall) = 0.965 (set membership — Palladius-invariant);
|
||||
// - catastrophe screen: 方源/蛊/蛊师 at ranks 0/1/2 (invariant), 古月 ∈ top-50 (rank 22 in default B —
|
||||
// the +56 Palladius bonus is dropped; the SCREEN still passes, exact rank is not a product value).
|
||||
//
|
||||
// It is DATA-GATED: the jieba artifact + book records are OUT of git (CLAUDE.md), so the test SKIPS when
|
||||
// they are absent (CI / a fresh checkout) and runs on the stand (or when TM_MINER_PARITY=1 forces it,
|
||||
// failing loud if the data is missing). $0, deterministic — Python is the reference; a divergence means
|
||||
// the Go port is wrong (fix Go), never the reference.
|
||||
const (
|
||||
minerParityContrast = "/home/ubuntu/projects/textmachine/eval/exp16/data/jieba_dict_general_zh.txt"
|
||||
minerParityRecords = "/home/ubuntu/books/gu-zhenren/rerun/records.json"
|
||||
minerParitySeed = "/home/ubuntu/books/gu-zhenren/guzhenren-seed-v2.yaml"
|
||||
)
|
||||
|
||||
func TestMinerFullBookParity(t *testing.T) {
|
||||
force := os.Getenv("TM_MINER_PARITY") == "1"
|
||||
for _, p := range []string{minerParityContrast, minerParityRecords, minerParitySeed} {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
if force {
|
||||
t.Fatalf("TM_MINER_PARITY=1 but required data is missing: %s", p)
|
||||
}
|
||||
t.Skipf("stand data absent (%s) — skipping full-book parity (set TM_MINER_PARITY=1 to force)", p)
|
||||
}
|
||||
}
|
||||
|
||||
// Contrast (jieba dict).
|
||||
cf, err := os.Open(minerParityContrast)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer cf.Close()
|
||||
contrast, err := LoadContrast(cf)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Chunks (records.json → normalized miner chunks).
|
||||
raw, err := os.ReadFile(minerParityRecords)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var recs []struct {
|
||||
Chapter int `json:"chapter"`
|
||||
ChunkIdx int `json:"chunk_idx"`
|
||||
Source string `json:"source"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &recs); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
chunks := make([]MinerChunk, 0, len(recs))
|
||||
for _, r := range recs {
|
||||
chunks = append(chunks, MinerChunk{Chapter: r.Chapter, ChunkIdx: r.ChunkIdx, NSource: normalizeSourceKey(r.Source)})
|
||||
}
|
||||
|
||||
// GT (the seed, parsed by the REAL loader — a divergence would fail seed-lint too).
|
||||
gt, err := loadGlossarySeed(minerParitySeed)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
mr := mineDetect(chunks, contrast, frozenMinerConfig())
|
||||
|
||||
// 1) candidate SET count.
|
||||
n := len(mr.ranked)
|
||||
if n != 13618 {
|
||||
t.Errorf("candidate SET count = %d, want 13618 (Palladius-invariant)", n)
|
||||
}
|
||||
|
||||
// 2) catastrophe screen.
|
||||
rank := map[string]int{}
|
||||
for i, c := range mr.ranked {
|
||||
rank[c.Src] = i
|
||||
}
|
||||
catExpect := map[string]int{"方源": 0, "蛊": 1, "蛊师": 2}
|
||||
for src, want := range catExpect {
|
||||
if got, ok := rank[normalizeSourceKey(src)]; !ok || got != want {
|
||||
t.Errorf("catastrophe rank[%s] = %d (present=%v), want %d (Palladius-invariant)", src, got, ok, want)
|
||||
}
|
||||
}
|
||||
if got, ok := rank[normalizeSourceKey("古月")]; !ok || got >= 50 {
|
||||
t.Errorf("catastrophe rank[古月] = %d (present=%v), want ∈top-50 (default B ≈22)", got, ok)
|
||||
} else {
|
||||
t.Logf("古月 rank = %d (default B — Palladius sub-channel dropped; reference A3 = 21)", got)
|
||||
}
|
||||
|
||||
// 3) recall@proposed (overall): fraction of GT entities with any normalized surface in the SET.
|
||||
candSet := make(map[string]bool, n)
|
||||
for _, c := range mr.ranked {
|
||||
candSet[c.Src] = true
|
||||
}
|
||||
hits := 0
|
||||
for _, e := range gt {
|
||||
surfaces := append([]string{e.Src}, aliasStrings(e.Aliases)...)
|
||||
for _, s := range surfaces {
|
||||
if nk := normalizeSourceKey(s); nk != "" && candSet[nk] {
|
||||
hits++
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
recall := float64(hits) / float64(len(gt))
|
||||
if recall < 0.960 || recall > 0.970 {
|
||||
t.Errorf("recall@proposed = %.4f (%d/%d), want ≈0.965", recall, hits, len(gt))
|
||||
}
|
||||
t.Logf("PARITY: n=%d catastrophe{方源:%d 蛊:%d 蛊师:%d 古月:%d} recall@proposed=%.4f (%d/%d GT)",
|
||||
n, rank[normalizeSourceKey("方源")], rank[normalizeSourceKey("蛊")], rank[normalizeSourceKey("蛊师")],
|
||||
rank[normalizeSourceKey("古月")], recall, hits, len(gt))
|
||||
}
|
||||
354
backend/internal/pipeline/miner_patterns.go
Normal file
354
backend/internal/pipeline/miner_patterns.go
Normal file
|
|
@ -0,0 +1,354 @@
|
|||
package pipeline
|
||||
|
||||
// miner_patterns.go: the V-C pattern channels (WS3 — a Go port of exp16 patterns.py). A language×genre
|
||||
// pattern pack for zh (§B5 plugin P3, universal set only — owner decision 18.07: genre packs are NOT
|
||||
// built). Each channel proposes TYPED candidates (name/title/place/term) with evidence, closing V-A's
|
||||
// frequency-blind classes (rare surname-anchored names, rank/grade titles, one-off realia). Versioned
|
||||
// data (surnames / title affixes / topo suffixes) + a book-adaptive productive-morphology detector
|
||||
// (channel 4 — auto-detects the domain formant, NOT hardcoded 蛊). Pure and deterministic.
|
||||
|
||||
// minerPackVersion is the pattern-inventory version (patterns.PACK_VERSION).
|
||||
const minerPackVersion = "zh-universal-v1"
|
||||
|
||||
// surnamesSingle is the 百家姓 single-char subset + this book's cast (patterns.SURNAMES_SINGLE, with 凝
|
||||
// discarded — it is not a surname). surnamesCompound are the explicit two-char compound surnames.
|
||||
var (
|
||||
surnamesSingle = buildRuneSet(surnamesSingleRaw)
|
||||
surnamesCompound = map[string]bool{
|
||||
"古月": true, "欧阳": true, "司马": true, "上官": true, "夏侯": true, "诸葛": true, "东方": true,
|
||||
"皇甫": true, "尉迟": true, "公孙": true, "慕容": true, "长孙": true, "宇文": true, "司徒": true,
|
||||
"鲜于": true, "南宫": true,
|
||||
}
|
||||
)
|
||||
|
||||
const surnamesSingleRaw = "赵钱孙李周吴郑王冯陈褚卫蒋沈韩杨朱秦尤许何吕施张孔曹严华金魏陶姜" +
|
||||
"戚谢邹喻柏水窦章云苏潘葛奚范彭郎鲁韦昌马苗凤花方俞任袁柳酆鲍史唐" +
|
||||
"费廉岑薛雷贺倪汤滕殷罗毕郝邬安常乐于时傅皮卞齐康伍余元卜顾孟平黄" +
|
||||
"和穆萧尹姚邵湛汪祁毛禹狄米贝明臧计伏成戴谈宋茅庞熊纪舒屈项祝董梁" +
|
||||
"杜阮蓝闵席季麻强贾路娄危江童颜郭梅盛林刁钟徐邱骆高夏蔡田樊胡凌霍" +
|
||||
"虞万支柯昝管卢莫经房裘缪干解应宗丁宣贲邓郁单杭洪包诸左石崔吉钮龚" +
|
||||
"白凝" // 凝 removed below via buildRuneSet's caller — kept literal for provenance parity
|
||||
|
||||
// buildRuneSet builds a rune set, then removes 凝 (patterns.SURNAMES_SINGLE.discard("凝")).
|
||||
func buildRuneSet(s string) map[rune]bool {
|
||||
m := map[rune]bool{}
|
||||
for _, r := range s {
|
||||
m[r] = true
|
||||
}
|
||||
delete(m, '凝')
|
||||
return m
|
||||
}
|
||||
|
||||
// Title / ordinal / topo / rank inventories (patterns.TITLE_SUFFIX/ORDINAL/TOPO_SUFFIX/GRADE_PREFIX/
|
||||
// NUMERAL/RANK_WORD). TITLE_PREFIX is defined in the reference but unused by pattern_candidates, so it
|
||||
// is omitted here.
|
||||
var (
|
||||
titleSuffix = []string{"公子", "大人", "长老", "前辈", "族长", "家老", "老祖", "祖师", "真人", "上人",
|
||||
"道人", "先生", "夫人", "娘子", "姑娘", "嬷嬷", "师傅", "师父", "掌门", "宗主",
|
||||
"少爷", "老爷", "小姐", "婆婆", "大娘", "大爷"}
|
||||
ordinalTitle = []string{"一代", "二代", "三代", "四代", "五代", "六代", "七代", "八代", "九代", "十代",
|
||||
"第一", "第二", "第三", "第四", "第五"}
|
||||
topoSuffix = buildRuneSet2("山寨村疆谷城门宗府洞峰岭河江湖海林原岛潭殿阁楼院堂祠")
|
||||
gradePrefix = buildRuneSet2("甲乙丙丁戊己庚辛壬癸")
|
||||
numeral = buildRuneSet2("一二三四五六七八九十零百千")
|
||||
rankWord = []string{"等", "转", "阶", "级", "重", "品", "段", "层"}
|
||||
)
|
||||
|
||||
// buildRuneSet2 builds a rune set with NO discard (topo/grade/numeral inventories).
|
||||
func buildRuneSet2(s string) map[rune]bool {
|
||||
m := map[rune]bool{}
|
||||
for _, r := range s {
|
||||
m[r] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// isSurnameStart returns the surname prefix (compound preferred) if s starts with one, else "" (patterns.
|
||||
// is_surname_start). s is a []rune (a Han run suffix).
|
||||
func isSurnameStart(s []rune) string {
|
||||
if len(s) >= 2 {
|
||||
if pre := string(s[:2]); surnamesCompound[pre] {
|
||||
return pre
|
||||
}
|
||||
}
|
||||
if len(s) >= 1 && surnamesSingle[s[0]] {
|
||||
return string(s[0])
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// patternInfo is one candidate's channel-proposed types + evidence, in first-seen (deterministic) order.
|
||||
type patternInfo struct {
|
||||
types []string
|
||||
evidence []string
|
||||
}
|
||||
|
||||
// formantInfo describes a detected productive-morphology formant (patterns.detect_formants entry).
|
||||
type formantInfo struct {
|
||||
role string // suffix|prefix|both
|
||||
overRep float64
|
||||
nSuffix int
|
||||
nPrefix int
|
||||
}
|
||||
|
||||
// runeHasPrefixAt reports whether run[i:] starts with p (patterns' run.startswith(suf, i)).
|
||||
func runeHasPrefixAt(run []rune, i int, p []rune) bool {
|
||||
if i+len(p) > len(run) {
|
||||
return false
|
||||
}
|
||||
for k := range p {
|
||||
if run[i+k] != p[k] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// formantType maps a formant char to a candidate type (patterns.formant_type): topo→place, 等/转/阶→
|
||||
// title, else term.
|
||||
func formantType(c rune) string {
|
||||
if topoSuffix[c] {
|
||||
return "place"
|
||||
}
|
||||
if c == '等' || c == '转' || c == '阶' {
|
||||
return "title"
|
||||
}
|
||||
return "term"
|
||||
}
|
||||
|
||||
// bookCharFreq counts Han chars over the normalized sources (patterns.book_char_freq).
|
||||
func bookCharFreq(chunks []MinerChunk) map[rune]int {
|
||||
bc := map[rune]int{}
|
||||
for _, c := range chunks {
|
||||
for _, r := range c.NSource {
|
||||
if isMinerHan(r) {
|
||||
bc[r]++
|
||||
}
|
||||
}
|
||||
}
|
||||
return bc
|
||||
}
|
||||
|
||||
// detectFormants auto-detects the productive DOMAIN formants (patterns.detect_formants): a Han char that
|
||||
// binds (suffix OR prefix) with ≥ minPartners DISTINCT content morphemes among candidates AND is
|
||||
// over-represented in-book vs general zh (over_rep ≥ minOverRep). Over-representation — NOT char-rarity
|
||||
// — isolates 蛊/窍/虫 while rejecting common chars 师/花/等. Deterministic (per-char, order-independent).
|
||||
func detectFormants(chunks []MinerChunk, candFreq map[string]int, contrast *Contrast, minPartners int, minOverRep float64) map[rune]formantInfo {
|
||||
bc := bookCharFreq(chunks)
|
||||
totBook := 0
|
||||
for _, v := range bc {
|
||||
totBook += v
|
||||
}
|
||||
if totBook == 0 {
|
||||
totBook = 1
|
||||
}
|
||||
suf := map[rune]map[string]bool{}
|
||||
pre := map[rune]map[string]bool{}
|
||||
addPartner := func(m map[rune]map[string]bool, key rune, part string) {
|
||||
if m[key] == nil {
|
||||
m[key] = map[string]bool{}
|
||||
}
|
||||
m[key][part] = true
|
||||
}
|
||||
for a := range candFreq {
|
||||
ar := []rune(a)
|
||||
if len(ar) < 2 {
|
||||
continue
|
||||
}
|
||||
addPartner(suf, ar[len(ar)-1], string(ar[:len(ar)-1]))
|
||||
addPartner(pre, ar[0], string(ar[1:]))
|
||||
}
|
||||
chars := map[rune]bool{}
|
||||
for c := range suf {
|
||||
chars[c] = true
|
||||
}
|
||||
for c := range pre {
|
||||
chars[c] = true
|
||||
}
|
||||
formants := map[rune]formantInfo{}
|
||||
for c := range chars {
|
||||
pBook := float64(bc[c]) / float64(totBook)
|
||||
overRep := contrast.charOverRep(c, pBook)
|
||||
if overRep < minOverRep {
|
||||
continue
|
||||
}
|
||||
ns, npr := len(suf[c]), len(pre[c])
|
||||
role := ""
|
||||
if ns >= minPartners {
|
||||
role = "suffix"
|
||||
}
|
||||
if npr >= minPartners {
|
||||
if role != "" {
|
||||
role = "both"
|
||||
} else {
|
||||
role = "prefix"
|
||||
}
|
||||
}
|
||||
if role != "" {
|
||||
formants[c] = formantInfo{role: role, overRep: overRep, nSuffix: ns, nPrefix: npr}
|
||||
}
|
||||
}
|
||||
return formants
|
||||
}
|
||||
|
||||
// patternCandidates produces the typed pattern candidates from the source (patterns.pattern_candidates).
|
||||
// It operates over the ALREADY-normalized source and may propose candidates BELOW the V-A freq floor
|
||||
// (that is the point — patterns close the rare-term blind spot). Returns the candidate→info map and the
|
||||
// detected formants. Deterministic (fixed chunk/run/position iteration; add() preserves first-seen order).
|
||||
func patternCandidates(chunks []MinerChunk, candFreq map[string]int, contrast *Contrast, minPartners int, minOverRep float64) (map[string]*patternInfo, map[rune]formantInfo) {
|
||||
out := map[string]*patternInfo{}
|
||||
add := func(cand []rune, typ, ev string) {
|
||||
cn := normalizeSourceKey(string(cand))
|
||||
if cn == "" || !minerHanOnly(cn) {
|
||||
return
|
||||
}
|
||||
info := out[cn]
|
||||
if info == nil {
|
||||
info = &patternInfo{}
|
||||
out[cn] = info
|
||||
}
|
||||
if !containsStr(info.types, typ) {
|
||||
info.types = append(info.types, typ)
|
||||
}
|
||||
if !containsStr(info.evidence, ev) {
|
||||
info.evidence = append(info.evidence, ev)
|
||||
}
|
||||
}
|
||||
|
||||
titleSufR := toRuneSlices(titleSuffix)
|
||||
ordinalR := toRuneSlices(ordinalTitle)
|
||||
rankR := toRuneSlices(rankWord)
|
||||
|
||||
for _, c := range chunks {
|
||||
for _, run := range hanRuns(c.NSource) {
|
||||
L := len(run)
|
||||
for i := 0; i < L; i++ {
|
||||
// (1) surname anchor: surname + 1–2 Han given-name window → name
|
||||
if sn := isSurnameStart(run[i:]); sn != "" {
|
||||
base := i + len([]rune(sn))
|
||||
for _, gl := range [2]int{1, 2} {
|
||||
if base+gl <= L {
|
||||
full := run[i : base+gl]
|
||||
if len(full) >= 2 && len(full) <= 4 {
|
||||
add(full, "name", "surname:"+sn)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// (2) title suffix: content + suffix → title
|
||||
for si, suf := range titleSufR {
|
||||
if runeHasPrefixAt(run, i, suf) {
|
||||
for _, left := range [4]int{3, 2, 1, 0} {
|
||||
if i-left >= 0 {
|
||||
cand := run[i-left : i+len(suf)]
|
||||
if len(cand) >= 2 && len(cand) <= 6 {
|
||||
add(cand, "title", "title_suffix:"+titleSuffix[si])
|
||||
}
|
||||
}
|
||||
}
|
||||
add([]rune(titleSuffix[si]), "title", "title_bare:"+titleSuffix[si])
|
||||
}
|
||||
}
|
||||
// (2b) ordinal + title (四代族长-class) → title
|
||||
for oi, od := range ordinalR {
|
||||
if runeHasPrefixAt(run, i, od) {
|
||||
for si, suf := range titleSufR {
|
||||
end := i + len(od)
|
||||
if runeHasPrefixAt(run, end, suf) {
|
||||
add(run[i:end+len(suf)], "title", "ordinal_title:"+ordinalTitle[oi]+"+"+titleSuffix[si])
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// (2c) rank/grade compositional: (numeral|grade-prefix) + rank-word → title
|
||||
for ri, rw := range rankR {
|
||||
if runeHasPrefixAt(run, i, rw) && i >= 1 {
|
||||
left := run[i-1]
|
||||
if numeral[left] || gradePrefix[left] {
|
||||
add(run[i-1:i+len(rw)], "title", "rank_grade:"+string(left)+"+"+rankWord[ri])
|
||||
}
|
||||
}
|
||||
}
|
||||
// (3) topo suffix: 1–3 Han base + topo char → place
|
||||
if topoSuffix[run[i]] && i >= 1 {
|
||||
for _, left := range [3]int{3, 2, 1} {
|
||||
if i-left >= 0 {
|
||||
cand := run[i-left : i+1]
|
||||
if len(cand) >= 2 && len(cand) <= 4 {
|
||||
add(cand, "place", "topo_suffix:"+string(run[i]))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// (4) productive morphology: for each detected formant, propose all binding n-grams.
|
||||
formants := detectFormants(chunks, candFreq, contrast, minPartners, minOverRep)
|
||||
for _, c := range chunks {
|
||||
for _, run := range hanRuns(c.NSource) {
|
||||
L := len(run)
|
||||
for i := 0; i < L; i++ {
|
||||
info, ok := formants[run[i]]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
typ := formantType(run[i])
|
||||
if info.role == "suffix" || info.role == "both" {
|
||||
for _, left := range [3]int{3, 2, 1} {
|
||||
if i-left >= 0 {
|
||||
cand := run[i-left : i+1]
|
||||
if len(cand) >= 2 && len(cand) <= 4 {
|
||||
add(cand, typ, "formant_suffix:"+string(run[i]))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if info.role == "prefix" || info.role == "both" {
|
||||
for _, r := range [2]int{2, 3} {
|
||||
if i+r <= L {
|
||||
cand := run[i : i+r]
|
||||
if len(cand) >= 2 && len(cand) <= 4 {
|
||||
add(cand, typ, "formant_prefix:"+string(run[i]))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, formants
|
||||
}
|
||||
|
||||
// minerHanOnly reports whether s is non-empty and all Han (exp16_common.han_only).
|
||||
func minerHanOnly(s string) bool {
|
||||
if s == "" {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if !isMinerHan(r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// toRuneSlices converts a []string inventory to []([]rune) once (avoids re-decoding per scan position).
|
||||
func toRuneSlices(ss []string) [][]rune {
|
||||
out := make([][]rune, len(ss))
|
||||
for i, s := range ss {
|
||||
out[i] = []rune(s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// containsStr reports whether ss contains s.
|
||||
func containsStr(ss []string, s string) bool {
|
||||
for _, x := range ss {
|
||||
if x == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
224
backend/internal/pipeline/miner_substrate.go
Normal file
224
backend/internal/pipeline/miner_substrate.go
Normal file
|
|
@ -0,0 +1,224 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"io"
|
||||
"math"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// miner_substrate.go: the deterministic bank-mining substrate (WS3, слой 4 — a Go port of the FROZEN
|
||||
// exp16 miner-v1: exp16_common.py). $0, no LLM, no embeddings. It provides the candidate generator (Han
|
||||
// char n-grams 1–6 over the memnorm-normalized source, NO word segmentation — §B1/Qiu&Zhang), the
|
||||
// general-zh contrast corpus (a versioned jieba word-freq artifact), overlapping occurrence counting
|
||||
// (Aho-Corasick-equivalent WITHOUT the hot-path suppressContained — D24.2), and the frequency strata.
|
||||
//
|
||||
// It reuses the VALIDATED normalizeSourceKey (memnorm.go, byte-faithful to the eval mem_select port), so
|
||||
// a candidate key and a seed/GT key live in the SAME normalized space (trad→simp, NFKC, kana-fold, lower)
|
||||
// — comparable to the jieba dict (simplified zh). The whole detector (this + miner_detect/patterns/arms)
|
||||
// is the WHICH channel; the dst (WHAT) is delivered by the banknote (WS4), never by co-occurrence
|
||||
// (canon-recovery 0.20–0.68 < 70% — research/20). Determinism is load-bearing: no time/rand, fixed
|
||||
// iteration; the mined delta is a pure function of (chunks, contrast, config).
|
||||
|
||||
// Miner n-gram bounds and the V-A frequency floor (exp16_common: NGRAM_MIN/MAX, FREQ_FLOOR).
|
||||
const (
|
||||
minerNgramMin = 1
|
||||
minerNgramMax = 6
|
||||
minerFreqFloor = 3 // §A1/§B1: below this V-A does not score (a conscious blind spot patterns close)
|
||||
)
|
||||
|
||||
// MinerChunk is one source chunk for mining: its chapter and the memnorm-normalized zh source. The
|
||||
// caller normalizes once (normalizeSourceKey) so the candidate space matches the glossary/GT space.
|
||||
type MinerChunk struct {
|
||||
Chapter int
|
||||
ChunkIdx int
|
||||
NSource string // normalizeSourceKey(raw source)
|
||||
}
|
||||
|
||||
// isMinerHan reports whether r is a Han ideograph — the candidate alphabet (§B1). Mirrors the Python
|
||||
// reference's \p{Han}: candidate n-grams are runs of Han characters only.
|
||||
func isMinerHan(r rune) bool { return unicode.Is(unicode.Han, r) }
|
||||
|
||||
// hanRuns returns the maximal runs of Han characters in normalized text (each run a []rune slice so
|
||||
// n-gram slicing is by CHARACTER, exactly like the Python str slicing).
|
||||
func hanRuns(ntext string) [][]rune {
|
||||
var runs [][]rune
|
||||
var cur []rune
|
||||
for _, r := range ntext {
|
||||
if isMinerHan(r) {
|
||||
cur = append(cur, r)
|
||||
} else if len(cur) > 0 {
|
||||
runs = append(runs, cur)
|
||||
cur = nil
|
||||
}
|
||||
}
|
||||
if len(cur) > 0 {
|
||||
runs = append(runs, cur)
|
||||
}
|
||||
return runs
|
||||
}
|
||||
|
||||
// enumerateCandidates counts every Han n-gram (length minerNgramMin..minerNgramMax) with OVERLAPPING
|
||||
// counts across the chunk sources (exp16_common.enumerate_candidates). The annotation chunk is included
|
||||
// (blurb terms live there; the GT/seed filter handles the blurb rule separately). Returns the full
|
||||
// multiset (before the freq floor) — its total token mass is the weirdness denominator (N_book).
|
||||
func enumerateCandidates(chunks []MinerChunk) (freq map[string]int, nBook int) {
|
||||
freq = make(map[string]int, 1<<16)
|
||||
for _, c := range chunks {
|
||||
for _, run := range hanRuns(c.NSource) {
|
||||
L := len(run)
|
||||
for n := minerNgramMin; n <= minerNgramMax; n++ {
|
||||
for i := 0; i+n <= L; i++ {
|
||||
freq[string(run[i:i+n])]++
|
||||
nBook++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return freq, nBook
|
||||
}
|
||||
|
||||
// countOccurrences totals the OVERLAPPING occurrences of a normalized needle across the chunk sources
|
||||
// (exp16_common.count_occurrences: pos advances by 1, so 重重 counts twice in 重重重). Suppress-free
|
||||
// (D24.2) — this is the occurrence axis, never the hot-path longest-match.
|
||||
func countOccurrences(needleNorm string, chunks []MinerChunk) int {
|
||||
if needleNorm == "" {
|
||||
return 0
|
||||
}
|
||||
total := 0
|
||||
for _, c := range chunks {
|
||||
hay := c.NSource
|
||||
for pos := 0; pos <= len(hay); {
|
||||
rel := strings.Index(hay[pos:], needleNorm)
|
||||
if rel < 0 {
|
||||
break
|
||||
}
|
||||
total++
|
||||
pos = pos + rel + 1 // overlap advance by ONE byte-position (mirrors Python find(needle, pos+1))
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// candidateChapters returns the set of chapters whose normalized source contains the candidate (for
|
||||
// since_ch = the first chapter of appearance, exp16_common.candidate_chapters). Deterministic.
|
||||
func candidateChapters(candNorm string, chunks []MinerChunk) map[int]bool {
|
||||
chs := map[int]bool{}
|
||||
if candNorm == "" {
|
||||
return chs
|
||||
}
|
||||
for _, c := range chunks {
|
||||
if strings.Contains(c.NSource, candNorm) {
|
||||
chs[c.Chapter] = true
|
||||
}
|
||||
}
|
||||
return chs
|
||||
}
|
||||
|
||||
// freqStratum buckets an occurrence count (exp16_common.freq_stratum).
|
||||
func freqStratum(f int) string {
|
||||
switch {
|
||||
case f >= 10:
|
||||
return "f>=10"
|
||||
case f >= minerFreqFloor:
|
||||
return "f3-9"
|
||||
default:
|
||||
return "f<3"
|
||||
}
|
||||
}
|
||||
|
||||
// --- general-zh contrast corpus (jieba dict.txt: "word freq POS" per line) ------------------------
|
||||
|
||||
// Contrast is the general-domain zh reference: normalized word frequencies + derived char frequencies,
|
||||
// used for the weirdness (domain-specificity) signal. A versioned artifact (the jieba 0.42.1 dict.txt,
|
||||
// SHA-pinned, kept OUT of git — reproducible from jieba). Load it once (miner-build time, $0).
|
||||
type Contrast struct {
|
||||
wordFreq map[string]int // normalized word → max freq (trad/simp collisions fold to the max)
|
||||
charFreq map[rune]int // char → summed freq across normalized words
|
||||
totalWord int64
|
||||
totalChar int64
|
||||
nDistinct int // len(charFreq), the Laplace |distinct| term
|
||||
}
|
||||
|
||||
// LoadContrast reads a jieba-style "word freq POS" corpus from r and builds the contrast model. Each
|
||||
// word is normalized into the candidate space (normalizeSourceKey) and, on a normalization collision,
|
||||
// the MAX frequency is kept (mirrors exp16_common.Contrast). Malformed / non-integer-freq lines are
|
||||
// skipped, exactly like the reference.
|
||||
func LoadContrast(r io.Reader) (*Contrast, error) {
|
||||
c := &Contrast{wordFreq: map[string]int{}, charFreq: map[rune]int{}}
|
||||
sc := bufio.NewScanner(r)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 1<<20)
|
||||
for sc.Scan() {
|
||||
fields := strings.Fields(sc.Text())
|
||||
if len(fields) < 2 {
|
||||
continue
|
||||
}
|
||||
w := fields[0]
|
||||
fr, err := strconv.Atoi(fields[1])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
wn := normalizeSourceKey(w)
|
||||
if wn == "" {
|
||||
continue
|
||||
}
|
||||
if prev, ok := c.wordFreq[wn]; !ok || fr > prev {
|
||||
c.wordFreq[wn] = fr
|
||||
}
|
||||
c.totalWord += int64(fr)
|
||||
for _, ch := range wn {
|
||||
c.charFreq[ch] += fr
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, v := range c.charFreq {
|
||||
c.totalChar += int64(v)
|
||||
}
|
||||
c.nDistinct = len(c.charFreq)
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// wordRel is P_general(ngram) treated as a whole word (0 for OOV).
|
||||
func (c *Contrast) wordRel(ngramNorm string) float64 {
|
||||
if c.totalWord == 0 {
|
||||
return 0
|
||||
}
|
||||
return float64(c.wordFreq[ngramNorm]) / float64(c.totalWord)
|
||||
}
|
||||
|
||||
// charIndepRel is the expected P_general(ngram) under char-independence with Laplace smoothing
|
||||
// (exp16_common.char_indep_rel): pc = (cf+1)/(total_char+|distinct|), product over chars.
|
||||
func (c *Contrast) charIndepRel(ngramNorm string) float64 {
|
||||
denom := float64(c.totalChar) + float64(c.nDistinct)
|
||||
if denom == 0 {
|
||||
return 0
|
||||
}
|
||||
p := 1.0
|
||||
for _, ch := range ngramNorm {
|
||||
pc := float64(c.charFreq[ch]+1) / denom
|
||||
p *= pc
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// charOverRep is p_book(char)/p_general(char) — the domain over-representation used by the formant
|
||||
// detector (patterns). p_general uses the same Laplace denominator as charIndepRel.
|
||||
func (c *Contrast) charOverRep(ch rune, pBook float64) float64 {
|
||||
denom := float64(c.totalChar) + float64(c.nDistinct)
|
||||
if denom == 0 {
|
||||
return 0
|
||||
}
|
||||
pGen := float64(c.charFreq[ch]+1) / denom
|
||||
if pGen == 0 {
|
||||
return 0
|
||||
}
|
||||
return pBook / pGen
|
||||
}
|
||||
|
||||
// lengthMult is the c-value g(L)=log2(L+1) length multiplier (miner_detect uses it; kept here beside
|
||||
// the substrate math). §B1: NOT log2(L) — that zeros |a|=1 (蛊/转) and breaks the catastrophe screen.
|
||||
func lengthMult(L int) float64 { return math.Log2(float64(L) + 1) }
|
||||
337
backend/internal/pipeline/miner_test.go
Normal file
337
backend/internal/pipeline/miner_test.go
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"textmachine/backend/internal/store"
|
||||
)
|
||||
|
||||
// miner_test.go: synthetic, in-repo unit fixtures for the WS3 default-B miner (no external book data —
|
||||
// the full-book Go↔Python parity is the data-gated TestMinerFullBookParity). Each fixture pins one
|
||||
// load-bearing algorithm piece: the c-value length-multiplier (蛊/转 non-degenerate), subsumption α, the
|
||||
// pattern channels, is_palladius_token, the alias tier-1 rules (the plan's required §3(д) fixtures), the
|
||||
// emission filters, determinism, and the Source:"mined" write path.
|
||||
|
||||
// mchunks builds miner chunks from raw zh strings (one chunk per chapter, normalized like the runner).
|
||||
func mchunks(sources ...string) []MinerChunk {
|
||||
out := make([]MinerChunk, len(sources))
|
||||
for i, s := range sources {
|
||||
out[i] = MinerChunk{Chapter: i + 1, ChunkIdx: 0, NSource: normalizeSourceKey(s)}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// smallContrast is a tiny general-zh reference: common function/name chars carry high freq; the domain
|
||||
// formant 蛊 is deliberately ABSENT so its book over-representation is high (the formant signal).
|
||||
func smallContrast(t *testing.T) *Contrast {
|
||||
t.Helper()
|
||||
const data = "的 100000 uj\n是 80000 v\n人 50000 n\n出 9000 v\n现 9000 v\n" +
|
||||
"月 4000 n\n光 3800 n\n希 900 n\n望 3000 v\n影 800 n\n在 20000 p\n此 1200 r\n" +
|
||||
"古 400 nr\n方 900 nr\n源 300 n\n族 500 n\n长 6000 a\n青 700 n\n茅 20 n\n山 5000 n\n" +
|
||||
"四 8000 m\n代 3000 n\n甲 300 n\n等 12000 u\n正 4000 a\n"
|
||||
c, err := LoadContrast(strings.NewReader(data))
|
||||
if err != nil {
|
||||
t.Fatalf("LoadContrast: %v", err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func TestMinerLengthMultNonDegenerate(t *testing.T) {
|
||||
// §B1: g(L)=log2(L+1), NOT log2(L) — a single-char candidate (蛊/转) must keep a NON-ZERO length
|
||||
// multiplier, else the catastrophe screen zeros out and 蛊 sinks to rank ~4753.
|
||||
if got := lengthMult(1); got != 1.0 {
|
||||
t.Fatalf("lengthMult(1) = %v, want 1.0 (log2(2)); log2(1)=0 would degenerate |a|=1 candidates", got)
|
||||
}
|
||||
if lengthMult(2) <= lengthMult(1) {
|
||||
t.Fatalf("length multiplier must be monotone in L")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMinerCValueSingleCharNested(t *testing.T) {
|
||||
// 蛊 (len 1) nested in 蛊师 (len 2): its c-value is g(1)·(f(蛊) − f(蛊师)) — computed, finite, and its
|
||||
// magnitude is non-degenerate because g(1)=1 (not 0). Pins the length-mult fix at the c-value layer.
|
||||
cf := map[string]int{"蛊": 10, "蛊师": 4}
|
||||
cv := computeCValue(cf)
|
||||
want := 1.0 * (10.0 - 4.0/1.0) // g(1)·(f(蛊) − avg over its 1 container)
|
||||
if got := cv["蛊"]; got != want {
|
||||
t.Fatalf("cval[蛊] = %v, want %v (g(1)=1 non-degenerate)", got, want)
|
||||
}
|
||||
if cv["蛊师"] != lengthMult(2)*4 { // non-nested → g(2)·f
|
||||
t.Fatalf("cval[蛊师] = %v, want %v", cv["蛊师"], lengthMult(2)*4)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMinerSubsumption(t *testing.T) {
|
||||
// 方源(559) survives 方源的(85) [ratio 0.15 < 0.80]; 花酒行(65) is dropped by 花酒行者(65) [ratio 1.0].
|
||||
cf := map[string]int{"方源": 559, "方源的": 85, "花酒行": 65, "花酒行者": 65}
|
||||
drop := subsumedCandidates(cf, minerSubsumeAlpha)
|
||||
if drop["方源"] {
|
||||
t.Fatalf("方源 must NOT be subsumed by 方源的 (85 < 0.8·559)")
|
||||
}
|
||||
if !drop["花酒行"] {
|
||||
t.Fatalf("花酒行 must be subsumed by 花酒行者 (65 >= 0.8·65)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMinerPatternChannels(t *testing.T) {
|
||||
chunks := mchunks(
|
||||
"古月方源走进青茅山。四代族长甲等弟子。", // surname+name, topo, ordinal_title, rank_grade
|
||||
)
|
||||
// candFreq is only needed for the formant channel; the structural channels scan the source directly.
|
||||
cf := map[string]int{}
|
||||
pats, _ := patternCandidates(chunks, cf, smallContrast(t), 3, 15.0)
|
||||
cases := []struct {
|
||||
src, typ string
|
||||
}{
|
||||
{"古月方源", "name"}, // surname anchor (compound 古月 + 方源)
|
||||
{"青茅山", "place"}, // topo suffix 山
|
||||
{"四代族长", "title"}, // ordinal_title 四代+族长
|
||||
{"甲等", "title"}, // rank_grade 甲+等
|
||||
{"族长", "title"}, // title_suffix / title_bare
|
||||
}
|
||||
for _, c := range cases {
|
||||
info, ok := pats[normalizeSourceKey(c.src)]
|
||||
if !ok {
|
||||
t.Errorf("pattern channel missed %q", c.src)
|
||||
continue
|
||||
}
|
||||
if !containsStr(info.types, c.typ) {
|
||||
t.Errorf("%q types = %v, want to include %q", c.src, info.types, c.typ)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMinerFormantDetection(t *testing.T) {
|
||||
// 蛊 binds ≥3 distinct morphemes among freq≥3 candidates AND is over-represented in-book vs the
|
||||
// contrast (absent there) → auto-detected as a suffix formant; a common char (的) is NOT.
|
||||
chunks := mchunks(strings.Repeat("月光蛊出现。希望蛊出现。月影蛊出现。", 3))
|
||||
va := runVA(chunks, smallContrast(t), minerFreqFloor, minerSubsumeAlpha)
|
||||
formants := detectFormants(chunks, va.candFreq, smallContrast(t), 3, 15.0)
|
||||
if _, ok := formants['蛊']; !ok {
|
||||
t.Fatalf("蛊 must be auto-detected as a productive formant, got %v", formantKeys(formants))
|
||||
}
|
||||
if _, ok := formants['的']; ok {
|
||||
t.Fatalf("的 (common char) must NOT be a formant")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMinerPalladiusToken(t *testing.T) {
|
||||
// Positives: transliterated Chinese-name shapes segment cleanly.
|
||||
for _, tok := range []string{"юань", "фан", "гу", "цзюй"} {
|
||||
if !isPalladiusToken(tok, 1) {
|
||||
t.Errorf("isPalladiusToken(%q) = false, want true (Palladius name shape)", tok)
|
||||
}
|
||||
}
|
||||
// Negative control (palladius.py): common Russian words are NOT Palladius tokens.
|
||||
for _, tok := range []string{"человек", "который", "деревня", "камень"} {
|
||||
if isPalladiusToken(tok, 1) {
|
||||
t.Errorf("isPalladiusToken(%q) = true, want false (common ru word)", tok)
|
||||
}
|
||||
}
|
||||
// The «найти»=най+ти FALSE-POSITIVE the pymorphy3 is_name_lemma gate exists to block: it IS
|
||||
// Palladius-shaped, which is exactly why default B drops the whole ru-side name confirmation channel
|
||||
// (no morphology backend to run the gate) rather than trust the token shape alone.
|
||||
if !isPalladiusToken("найти", 2) {
|
||||
t.Fatalf("найти must be Palladius-shaped (documents why default-B drops the ru name channel)")
|
||||
}
|
||||
}
|
||||
|
||||
// aliasSurf builds an aliasSurface for the rule tests.
|
||||
func aliasSurf(src, typ, gender, approvedDst string) aliasSurface {
|
||||
return aliasSurface{src: normalizeSourceKey(src), typ: typ, gender: gender, approvedDst: approvedDst}
|
||||
}
|
||||
|
||||
func TestMinerAliasRules(t *testing.T) {
|
||||
seedSurf := func(ss ...string) map[string]bool {
|
||||
m := map[string]bool{}
|
||||
for _, s := range ss {
|
||||
m[normalizeSourceKey(s)] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
hasIdent := func(edges []aliasEdge, a, b string) bool {
|
||||
an, bn := normalizeSourceKey(a), normalizeSourceKey(b)
|
||||
for _, e := range edges {
|
||||
if (e.a == an && e.b == bn) || (e.a == bn && e.b == an) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// R1 extension: 方源 (seed entity) ⊂ 古月方源 → identity (same person, fuller surface).
|
||||
t.Run("R1_extension", func(t *testing.T) {
|
||||
surf := map[string]aliasSurface{
|
||||
normalizeSourceKey("方源"): aliasSurf("方源", "name", "", ""),
|
||||
normalizeSourceKey("古月方源"): aliasSurf("古月方源", "name", "", ""),
|
||||
}
|
||||
ident, _, _ := proposeAliasEdges(surf, nil, seedSurf("方源"))
|
||||
if !hasIdent(ident, "方源", "古月方源") {
|
||||
t.Fatalf("方源 ⊂ 古月方源 must be an identity extension, got %+v", ident)
|
||||
}
|
||||
})
|
||||
|
||||
// Composite guard: 古月 (seed clan) + 族长 (seed title) = 古月族长 is a PHRASE, not an alias (blocks
|
||||
// the clan↔title chaining). But 古月 + 方源 (name) stays a fullname alias.
|
||||
t.Run("composite_guard", func(t *testing.T) {
|
||||
surf := map[string]aliasSurface{
|
||||
normalizeSourceKey("古月"): aliasSurf("古月", "name", "", ""),
|
||||
normalizeSourceKey("族长"): aliasSurf("族长", "title", "", ""),
|
||||
normalizeSourceKey("古月族长"): aliasSurf("古月族长", "title", "", ""),
|
||||
normalizeSourceKey("古月方源"): aliasSurf("古月方源", "name", "", ""),
|
||||
normalizeSourceKey("方源"): aliasSurf("方源", "name", "", ""),
|
||||
}
|
||||
seeds := seedSurf("古月", "族长", "方源")
|
||||
ident, _, weak := proposeAliasEdges(surf, nil, seeds)
|
||||
if hasIdent(ident, "古月", "古月族长") {
|
||||
t.Fatalf("古月+族长 (clan+title) must be a phrase, NOT an identity alias")
|
||||
}
|
||||
phraseFlagged := false
|
||||
for _, e := range weak {
|
||||
if e.kind == "phrase_not_alias" && e.a == normalizeSourceKey("古月") && e.b == normalizeSourceKey("古月族长") {
|
||||
phraseFlagged = true
|
||||
}
|
||||
}
|
||||
if !phraseFlagged {
|
||||
t.Fatalf("古月+族长 must be flagged phrase_not_alias, got weak=%+v", weak)
|
||||
}
|
||||
if !hasIdent(ident, "古月", "古月方源") {
|
||||
t.Fatalf("古月+方源 (clan+name) must stay a fullname identity alias, got %+v", ident)
|
||||
}
|
||||
})
|
||||
|
||||
// R4-iii/v: 族长 (approved «глава клана») ⊂ 四代族长 (approved «четвёртый глава клана») — DIFFERENT
|
||||
// approved dst → different entities EVEN under containment (D38 §3). No identity edge.
|
||||
t.Run("R4_different_approved_dst", func(t *testing.T) {
|
||||
surf := map[string]aliasSurface{
|
||||
normalizeSourceKey("族长"): aliasSurf("族长", "title", "", "глава клана"),
|
||||
normalizeSourceKey("四代族长"): aliasSurf("四代族长", "title", "", "четвёртый глава клана"),
|
||||
}
|
||||
ident, _, _ := proposeAliasEdges(surf, nil, seedSurf("族长", "四代族长"))
|
||||
if hasIdent(ident, "族长", "四代族长") {
|
||||
t.Fatalf("族长 vs 四代族长 with different approved dst must NOT be identity, got %+v", ident)
|
||||
}
|
||||
})
|
||||
|
||||
// R4-ii: different confirmed gender blocks identity; hidden never blocks.
|
||||
t.Run("R4_gender", func(t *testing.T) {
|
||||
surf := map[string]aliasSurface{
|
||||
normalizeSourceKey("方源"): aliasSurf("方源", "name", "male", ""),
|
||||
normalizeSourceKey("方源儿"): aliasSurf("方源儿", "name", "female", ""),
|
||||
}
|
||||
ident, _, _ := proposeAliasEdges(surf, nil, seedSurf("方源"))
|
||||
if hasIdent(ident, "方源", "方源儿") {
|
||||
t.Fatalf("different confirmed gender must block identity, got %+v", ident)
|
||||
}
|
||||
})
|
||||
|
||||
// R4-i / R2: same surname, DIFFERENT given names → family, not identity (方源 vs 方正, surname 方).
|
||||
t.Run("R2_family_R4i", func(t *testing.T) {
|
||||
surf := map[string]aliasSurface{
|
||||
normalizeSourceKey("方源"): aliasSurf("方源", "name", "", ""),
|
||||
normalizeSourceKey("方正"): aliasSurf("方正", "name", "", ""),
|
||||
}
|
||||
ident, family, _ := proposeAliasEdges(surf, nil, seedSurf("方源", "方正"))
|
||||
if hasIdent(ident, "方源", "方正") {
|
||||
t.Fatalf("same surname + different given names must be family, NOT identity")
|
||||
}
|
||||
if len(family) == 0 {
|
||||
t.Fatalf("方源 ~ 方正 must be a family edge, got none")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestMinerEmissionFilters(t *testing.T) {
|
||||
subsumed := map[string]bool{"花酒行": true}
|
||||
seeds := map[string]bool{normalizeSourceKey("方源"): true}
|
||||
cases := []struct {
|
||||
name string
|
||||
c ScoredCand
|
||||
want bool
|
||||
}{
|
||||
{"typed name freq5", ScoredCand{Src: "龙公", Types: []string{"name"}, Freq: 5}, true},
|
||||
{"below freq floor", ScoredCand{Src: "龙公", Types: []string{"name"}, Freq: 4}, false},
|
||||
{"untyped", ScoredCand{Src: "什么", Types: []string{"term"}, Freq: 20}, false},
|
||||
{"subsumed", ScoredCand{Src: "花酒行", Types: []string{"place"}, Freq: 9}, false},
|
||||
{"single char", ScoredCand{Src: "蛊", Types: []string{"term", "name"}, Freq: 99}, false},
|
||||
{"fragment", ScoredCand{Src: normalizeSourceKey("方源心"), Types: []string{"name"}, Freq: 9}, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := emissionEligible(tc.c, subsumed, seeds); got != tc.want {
|
||||
t.Errorf("%s: emissionEligible = %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMineBankDeterministicAndNonSeed(t *testing.T) {
|
||||
chunks := mchunks(strings.Repeat("龙公来到青茅山。龙公很强。青茅山很高。", 4))
|
||||
seed := []store.GlossaryEntry{{Src: "青茅山", Dst: "гора Цинмао", Type: "place", Status: "approved", Source: "seed"}}
|
||||
cfg := frozenMinerConfig()
|
||||
a := MineBank(chunks, smallContrast(t), seed, cfg)
|
||||
b := MineBank(chunks, smallContrast(t), seed, cfg)
|
||||
if !reflect.DeepEqual(a, b) {
|
||||
t.Fatalf("MineBank must be deterministic byte-for-byte:\n a=%+v\n b=%+v", a, b)
|
||||
}
|
||||
// The seed surface 青茅山 must NOT be re-emitted as a new mined term (the curated entry owns it).
|
||||
for _, m := range a {
|
||||
if m.Src == normalizeSourceKey("青茅山") {
|
||||
t.Fatalf("MineBank re-emitted a seed surface as a new term: %+v", m)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMinedToCandidatesStampsMinedSource(t *testing.T) {
|
||||
mined := []MinedTerm{
|
||||
{Src: "龙公", Type: "name", SinceCh: 3, Freq: 7, Aliases: []string{"龙公子"}},
|
||||
{Src: "青茅山", Type: "place", SinceCh: 1, Freq: 9}, // covered by manual seed → skipped
|
||||
}
|
||||
manual := map[string]bool{"青茅山": true}
|
||||
got := minedToCandidates(mined, manual)
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("want 1 mined candidate (青茅山 skipped as manual), got %d", len(got))
|
||||
}
|
||||
e := got[0]
|
||||
if e.Source != "mined" {
|
||||
t.Fatalf("mined candidate Source = %q, want \"mined\" (NOT seed — else it moves base-bank-version)", e.Source)
|
||||
}
|
||||
if e.Status != "auto" || e.Dst != "" {
|
||||
t.Fatalf("mined candidate must be status=auto with no dst, got status=%q dst=%q", e.Status, e.Dst)
|
||||
}
|
||||
if e.SinceCh != 3 || e.Confidence != 7 {
|
||||
t.Fatalf("mined candidate metadata wrong: %+v", e)
|
||||
}
|
||||
if len(e.Aliases) != 1 || e.Aliases[0].Alias != "龙公子" || e.Aliases[0].AliasType != "mined" {
|
||||
t.Fatalf("mined alias wrong: %+v", e.Aliases)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMinedDeltaStaysBaseBankStable(t *testing.T) {
|
||||
// The mined delta (Source:"mined") must NOT move BaseVersion — the load-bearing «переоплата ОДНА»
|
||||
// invariant (§1в): a W1.5 mined-row addition moves only the enriched version. (Sibling of the
|
||||
// Block-A TestBaseEnrichedMemoryVersionSplit, here through the real mined-write path.)
|
||||
seed := []store.GlossaryEntry{{Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"}}
|
||||
base0 := materializeMemory(seed, false)
|
||||
mined := minedToCandidates([]MinedTerm{{Src: "龙公", Type: "name", SinceCh: 1, Freq: 5}}, map[string]bool{"方源": true})
|
||||
// A mined candidate is status=auto (not folded even into enriched when gate off) — promote it to
|
||||
// approved to exercise the split, as the owner sign would at W1.5.
|
||||
mined[0].Status = "approved"
|
||||
mined[0].Dst = "Лун Гун"
|
||||
enriched := append(append([]store.GlossaryEntry{}, seed...), mined...)
|
||||
bank1 := materializeMemory(enriched, false)
|
||||
if base0.BaseVersion() != bank1.BaseVersion() {
|
||||
t.Fatalf("base-bank-version moved on a Source:mined addition — snapshot_W1 would re-bill W1")
|
||||
}
|
||||
if base0.Version() == bank1.Version() {
|
||||
t.Fatalf("enriched version must move on a mined approved addition")
|
||||
}
|
||||
}
|
||||
|
||||
func formantKeys(m map[rune]formantInfo) []string {
|
||||
out := make([]string, 0, len(m))
|
||||
for r := range m {
|
||||
out = append(out, string(r))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
|
@ -111,6 +111,7 @@ type projectOpts struct {
|
|||
gatesYAML string // optional gates:/... block appended to pipeline.yaml
|
||||
glossarySeed string // optional glossary seed YAML content; "" = no glossary
|
||||
postcheckGate bool // enable the memory post-check hard gate (assumes gatesYAML is empty)
|
||||
banknote bool // enable the banknote channel gate (WS4; assumes gatesYAML/postcheckGate empty)
|
||||
}
|
||||
|
||||
func setupProjectOpts(t *testing.T, providerURL string, o projectOpts) string {
|
||||
|
|
@ -149,6 +150,9 @@ models:
|
|||
if o.postcheckGate {
|
||||
gatesBlock += "\ngates:\n glossary:\n postcheck_gate: true\n"
|
||||
}
|
||||
if o.banknote {
|
||||
gatesBlock += "\ngates:\n banknote:\n enabled: true\n"
|
||||
}
|
||||
writeFile(t, filepath.Join(dir, "pipeline.yaml"), fmt.Sprintf(`
|
||||
core: C1
|
||||
version: 1
|
||||
|
|
|
|||
66
backend/internal/pipeline/seedlint_test.go
Normal file
66
backend/internal/pipeline/seedlint_test.go
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"textmachine/backend/internal/store"
|
||||
)
|
||||
|
||||
// seedlint_test.go: WS3 (д) `tmctl seed-lint` — the dry-run of the REAL loadGlossarySeed fail-louds over
|
||||
// a seed YAML (the mined delta or a manual seed). Both directions: a clean seed lints OK; a defect fails
|
||||
// loud; and the emitted mined delta (MinedDeltaYAML) round-trips through the real loader with 0 fails.
|
||||
|
||||
func TestSeedLintCleanPasses(t *testing.T) {
|
||||
p := writeSeed(t, `
|
||||
terms:
|
||||
- src: 方源
|
||||
dst: Фан Юань
|
||||
type: name
|
||||
status: approved
|
||||
- src: 蛊
|
||||
status: auto
|
||||
`)
|
||||
if err := SeedLint(p); err != nil {
|
||||
t.Fatalf("a clean seed must lint OK, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedLintCatchesDefects(t *testing.T) {
|
||||
cases := []struct {
|
||||
name, yaml, want string
|
||||
}{
|
||||
{"approved without dst", "terms:\n - src: 方源\n status: approved\n", "non-empty dst"},
|
||||
{"unknown status", "terms:\n - src: 方源\n dst: X\n status: bogus\n", "status must be"},
|
||||
{"duplicate key", "terms:\n - src: 方源\n dst: A\n - src: 方源\n dst: B\n", "duplicate"},
|
||||
{"shared-key collision", "terms:\n - src: 老赵\n dst: Старина Чжао\n status: approved\n - src: 赵大\n dst: Чжао Да\n status: approved\n aliases:\n - alias: 老赵\n", "shared"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
p := writeSeed(t, tc.yaml)
|
||||
err := SeedLint(p)
|
||||
if err == nil {
|
||||
t.Fatalf("%s: seed-lint must fail loud", tc.name)
|
||||
}
|
||||
if !strings.Contains(err.Error(), tc.want) {
|
||||
t.Fatalf("%s: error %q must mention %q", tc.name, err.Error(), tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedLintEmittedMinedDelta(t *testing.T) {
|
||||
// The emitted mined delta must be loadable by the REAL loader (0 fail-louds) — the pre-condition for
|
||||
// the W1.5 reseed.
|
||||
chunks := mchunks(strings.Repeat("龙公来到青茅山。龙公很强。青茅山很高。", 4))
|
||||
seed := []store.GlossaryEntry{{Src: "青茅山", Dst: "гора Цинмао", Type: "place", Status: "approved", Source: "seed"}}
|
||||
mined := MineBank(chunks, smallContrast(t), seed, frozenMinerConfig())
|
||||
yamlStr, err := MinedDeltaYAML(mined)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := writeSeed(t, yamlStr)
|
||||
if err := SeedLint(p); err != nil {
|
||||
t.Fatalf("the emitted mined delta must lint clean (loadable), got:\n%v\n--- delta ---\n%s", err, yamlStr)
|
||||
}
|
||||
}
|
||||
|
|
@ -4,6 +4,8 @@ import (
|
|||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
|
||||
"textmachine/backend/internal/config"
|
||||
)
|
||||
|
||||
// snapshot.go: материализация snapshotID (§3.2/D5.2/D8) — контент-хеш всего, что
|
||||
|
|
@ -97,6 +99,28 @@ func (r *Runner) sanitizerSnapshot() sanitizerSnap {
|
|||
return s
|
||||
}
|
||||
|
||||
// banknoteSnap freezes the banknote channel (WS4 §4в, F9) inside the snapshot — the VERDICT-AXIS that
|
||||
// governs the resolved draft (the slice + the derived-checkpoint), mirroring sanitizerSnap. parser_version
|
||||
// is the split/parse algorithm version: bumping it re-resolves the stripped draft, so it must be a loud
|
||||
// --resnapshot even without a prompt edit (§4а point 6). DEVIATION-with-rationale from "mirror
|
||||
// sanitizerSnap exactly": it is folded via a POINTER with omitempty (nil when the channel is off) rather
|
||||
// than an unconditional {enabled:false} — so ADDING this optional feature does NOT invalidate every
|
||||
// banknote-OFF book (incl. the golden fixture), while ENABLING the channel or bumping the parser is still
|
||||
// a loud snapshot move (the substantive loudness point 6 requires). A snapshot re-pin is never billed for
|
||||
// a feature a book does not use.
|
||||
type banknoteSnap struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
ParserVersion string `json:"parser_version,omitempty"`
|
||||
}
|
||||
|
||||
// banknoteSnapshot returns the fold ONLY when the channel is enabled (nil otherwise → omitempty drops it).
|
||||
func (r *Runner) banknoteSnapshot() *banknoteSnap {
|
||||
if !r.Pipeline.Gates.Banknote.Enabled {
|
||||
return nil
|
||||
}
|
||||
return &banknoteSnap{Enabled: true, ParserVersion: bankParserVersion}
|
||||
}
|
||||
|
||||
// memoryVersion is the content-hash of the deterministically materialized injected
|
||||
// memory (the frozen APPROVED glossary rows + the normalization/matcher algorithm
|
||||
// versions), the memory component of the snapshot (D5.2/D8, F1 CLOSED). It is the
|
||||
|
|
@ -114,12 +138,65 @@ func (r *Runner) memoryVersion() string {
|
|||
return computeMemoryVersion(nil, r.Pipeline.Gates.Glossary.PostcheckGate)
|
||||
}
|
||||
|
||||
// snapshotID materializes the job-context snapshot (§3.2): brief_hash, chunker
|
||||
// version, context-assembly knobs, the memory version and the full stage plan
|
||||
// (model, prompt version + CONTENT hash, sampling, resolved capability).
|
||||
// Payload is rendered with a fixed field order — the id is a content hash, so
|
||||
// identical context re-upserts idempotently.
|
||||
// wave selects which pipeline wave a per-wave snapshot folds (WS1 §1в).
|
||||
type wave int
|
||||
|
||||
const (
|
||||
waveW1 wave = iota // DRAFT wave: translator-role stages + the BASE bank version (excl Source:mined)
|
||||
waveW2 // EDIT wave: non-translator stages + the ENRICHED bank version (all approved incl mined)
|
||||
)
|
||||
|
||||
// snapshotID materializes the BOOK-GLOBAL job-context snapshot (§3.2): brief_hash, chunker version,
|
||||
// context-assembly knobs, the (enriched) memory version and the FULL stage plan. It is the snapshot the
|
||||
// SEQUENTIAL driver pins every job to; the byte output is unchanged (buildSnapshotID over all stages +
|
||||
// the enriched memory version reproduces it exactly). Payload is a fixed-field-order content hash.
|
||||
func (r *Runner) snapshotID() (id, payload string, err error) {
|
||||
return r.buildSnapshotID(r.Pipeline.Stages, r.memoryVersion())
|
||||
}
|
||||
|
||||
// snapshotIDForWave computes the PER-WAVE snapshot (WS1 §1в, ADDITIVE — the wave executor residual pins
|
||||
// each wave's jobs to it; the sequential driver still uses snapshotID). W1 folds the DRAFT stages
|
||||
// (translator role) + the BASE bank version (Source∈{seed,ruby,auto}, EXCL mined); W2 folds the EDIT
|
||||
// stages (non-translator) + the ENRICHED bank version (all approved incl Source:mined). A W1.5 bank
|
||||
// enrichment (adding mined rows) moves ONLY the enriched version → ONLY snapshot_W2, keeping W1
|
||||
// checkpoints valid («переоплата ОДНА»). Byte-consistent with snapshotID's building — the SAME payload
|
||||
// struct, only the stage subset and the memory version differ.
|
||||
func (r *Runner) snapshotIDForWave(w wave) (id, payload string, err error) {
|
||||
memVer := r.memoryVersion() // enriched (W2)
|
||||
if w == waveW1 {
|
||||
memVer = r.baseMemoryVersion()
|
||||
}
|
||||
return r.buildSnapshotID(waveStages(r.Pipeline.Stages, w), memVer)
|
||||
}
|
||||
|
||||
// waveStages partitions the pipeline stages by role for a wave: W1 = the translator-role (draft) stages,
|
||||
// W2 = every non-translator (editor/other) stage. Order preserved. The prod C1 core is one draft + one
|
||||
// edit, so this yields [draft] and [edit] respectively.
|
||||
func waveStages(stages []config.Stage, w wave) []config.Stage {
|
||||
var out []config.Stage
|
||||
for _, st := range stages {
|
||||
isDraft := st.Role == roleTranslator
|
||||
if (w == waveW1) == isDraft {
|
||||
out = append(out, st)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// baseMemoryVersion is the DRAFT-wave (W1) memory component: the BASE bank version (excl Source:mined).
|
||||
// nil bank → the empty-materialization base hash (a stable constant), mirroring memoryVersion().
|
||||
func (r *Runner) baseMemoryVersion() string {
|
||||
if r.memory != nil {
|
||||
return r.memory.BaseVersion()
|
||||
}
|
||||
return computeMemoryVersionScoped(nil, r.Pipeline.Gates.Glossary.PostcheckGate, true)
|
||||
}
|
||||
|
||||
// buildSnapshotID materializes the snapshot payload for a given stage subset + memory version (§3.2).
|
||||
// Extracted from snapshotID so the book-global snapshot (all stages + enriched version) AND the two
|
||||
// per-wave snapshots (WS1 §1в) share one byte-exact builder — a stage subset / memory version is the
|
||||
// ONLY axis that differs. Fixed field order (a content hash).
|
||||
func (r *Runner) buildSnapshotID(stages []config.Stage, memVersion string) (id, payload string, err error) {
|
||||
type stageSnap struct {
|
||||
Name string `json:"name"`
|
||||
Role string `json:"role"`
|
||||
|
|
@ -238,6 +315,9 @@ func (r *Runner) snapshotID() (id, payload string, err error) {
|
|||
// but determines a checkpoint's RESOLVED verdict (a defect flips the chunk to flagged), so a
|
||||
// gate flip / rule edit is a loud --resnapshot. Folded only when enabled (sanitizerSnapshot).
|
||||
Sanitizer sanitizerSnap `json:"sanitizer"`
|
||||
// Banknote — the banknote channel (WS4). Verdict-axis (governs the sliced/derived draft), folded
|
||||
// only when ENABLED (nil→omitempty otherwise), so adding the feature never re-bills an off book.
|
||||
Banknote *banknoteSnap `json:"banknote,omitempty"`
|
||||
Stages []stageSnap `json:"stages"`
|
||||
}{
|
||||
BriefHash: r.Book.BriefHash(),
|
||||
|
|
@ -255,13 +335,14 @@ func (r *Runner) snapshotID() (id, payload string, err error) {
|
|||
},
|
||||
Segmentation: r.segmentationSnapshot(),
|
||||
RenderFormatVersion: renderFormatVersion,
|
||||
MemoryVersion: r.memoryVersion(),
|
||||
MemoryVersion: memVersion,
|
||||
PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate,
|
||||
Coverage: r.coverageSnapshot(),
|
||||
StyleCheckVersion: cheapGateVersion,
|
||||
Sanitizer: r.sanitizerSnapshot(),
|
||||
Banknote: r.banknoteSnapshot(),
|
||||
}
|
||||
for _, st := range r.Pipeline.Stages {
|
||||
for _, st := range stages {
|
||||
ss := stageSnap{
|
||||
Name: st.Name, Role: st.Role, Model: st.Model,
|
||||
PromptVersion: st.PromptVersion, PromptSHA256: r.templates[st.Name].SHA256,
|
||||
|
|
|
|||
71
backend/internal/pipeline/snapshot_wave_test.go
Normal file
71
backend/internal/pipeline/snapshot_wave_test.go
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"textmachine/backend/internal/store"
|
||||
)
|
||||
|
||||
// snapshot_wave_test.go: WS1 §1в per-wave snapshot mechanics (R1 ADDITIVE sub-step — the wave executor
|
||||
// itself is residual; this pins the snapshot machinery it will use, WITHOUT switching the driver). The
|
||||
// book-global snapshotID() is byte-unchanged (TestGolden proves it); these tests exercise the new
|
||||
// snapshotIDForWave.
|
||||
func TestSnapshotIDForWave(t *testing.T) {
|
||||
r := newRunner(t, setupProject(t, "http://127.0.0.1:1"))
|
||||
defer r.Close()
|
||||
|
||||
seed := []store.GlossaryEntry{
|
||||
{Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"},
|
||||
{Src: "古月", Dst: "Гу Юэ", Status: "approved", Source: "ruby"},
|
||||
}
|
||||
r.memory = materializeMemory(seed, false)
|
||||
|
||||
w1id, w1p, err := r.snapshotIDForWave(waveW1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w2id, w2p, err := r.snapshotIDForWave(waveW2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// W1 and W2 differ (different stage subset AND different bank version).
|
||||
if w1id == w2id {
|
||||
t.Fatalf("snapshot_W1 must differ from snapshot_W2 (stage subset + bank version)")
|
||||
}
|
||||
// The stage PARTITION: W1 folds the draft (translator) stage, W2 folds the edit (editor) stage.
|
||||
if !strings.Contains(w1p, `"name":"draft"`) || strings.Contains(w1p, `"name":"edit"`) {
|
||||
t.Fatalf("snapshot_W1 must fold ONLY the draft stage, payload: %s", w1p)
|
||||
}
|
||||
if !strings.Contains(w2p, `"name":"edit"`) || strings.Contains(w2p, `"name":"draft"`) {
|
||||
t.Fatalf("snapshot_W2 must fold ONLY the edit stage, payload: %s", w2p)
|
||||
}
|
||||
|
||||
// «Переоплата ОДНА» at the snapshot level (§1в): a W1.5 mined-approved addition moves ONLY W2.
|
||||
enriched := append(append([]store.GlossaryEntry{}, seed...), store.GlossaryEntry{
|
||||
Src: "蛊", Dst: "гу", Status: "approved", Source: "mined",
|
||||
})
|
||||
r.memory = materializeMemory(enriched, false)
|
||||
w1id2, _, _ := r.snapshotIDForWave(waveW1)
|
||||
w2id2, _, _ := r.snapshotIDForWave(waveW2)
|
||||
if w1id2 != w1id {
|
||||
t.Fatalf("snapshot_W1 moved on a mined-approved addition — W1 checkpoints would re-bill («переоплата ОДНА» broken)")
|
||||
}
|
||||
if w2id2 == w2id {
|
||||
t.Fatalf("snapshot_W2 did NOT move on a mined-approved addition — the edit wave would miss the mined canon")
|
||||
}
|
||||
}
|
||||
|
||||
// TestWaveStagesPartition pins the role partition independently of the runner.
|
||||
func TestWaveStagesPartition(t *testing.T) {
|
||||
r := newRunner(t, setupProject(t, "http://127.0.0.1:1"))
|
||||
defer r.Close()
|
||||
w1 := waveStages(r.Pipeline.Stages, waveW1)
|
||||
w2 := waveStages(r.Pipeline.Stages, waveW2)
|
||||
if len(w1) != 1 || w1[0].Role != roleTranslator {
|
||||
t.Fatalf("W1 must be the translator stage(s), got %+v", w1)
|
||||
}
|
||||
if len(w2) != 1 || w2[0].Role != roleEditor {
|
||||
t.Fatalf("W2 must be the editor stage(s), got %+v", w2)
|
||||
}
|
||||
}
|
||||
|
|
@ -99,6 +99,13 @@ func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, sn
|
|||
// inherited 2048/3291 and returned length.
|
||||
baseMaxTokens = r.applyModelFloor(baseMaxTokens, st.Model)
|
||||
|
||||
// Banknote (WS4 point 5): reserve room for the ≤12-line footnote block so a length cut hits the
|
||||
// block (tolerated), not the translation. Translator-only + channel-enabled; wire-affecting
|
||||
// (max_tokens ∈ request_hash) and reproducible on resume (banknoteSnap gates the enabled state).
|
||||
if st.Role == roleTranslator && r.Pipeline.Gates.Banknote.Enabled {
|
||||
baseMaxTokens += bankTokenBudget
|
||||
}
|
||||
|
||||
maxRegen := r.Pipeline.Retries.RegenerateBeforeEscalate
|
||||
if maxRegen < 0 {
|
||||
maxRegen = 0
|
||||
|
|
@ -167,6 +174,19 @@ func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, sn
|
|||
recovered := ""
|
||||
if disposition == DispOK {
|
||||
finalHash = last.reqHash
|
||||
// Banknote OK-path (WS4 point 8, the main code change): if the translator draft carried a
|
||||
// ⟦TM-BANK-v1⟧ block, commit the CLEANED draft as a $0 derived checkpoint and re-point final_hash
|
||||
// at it — extending the flagged→ok derived-export precedent to the OK path — so the editor + the
|
||||
// export read the cleaned draft on resume, never the raw footnote-bearing one. `last` is the
|
||||
// terminal (possibly escalated) attempt whose block was sliced, so this runs AFTER the escalation
|
||||
// resolve and consumes the escalated draft's stripped text (point 8's escalation-ordering note).
|
||||
if last.bankStripped != "" {
|
||||
dh, err := r.commitBanknoteExport(st, ch, job, last)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
finalHash = dh
|
||||
}
|
||||
} else if last.cls.Reason == FlagSanitizerStripped {
|
||||
// Cosmetic leak (D35.4a): COMMIT the cleaned remainder as a $0 derived export checkpoint, then
|
||||
// point final_hash at it — so the standard final_hash→checkpoint.response_text export
|
||||
|
|
@ -211,6 +231,7 @@ func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, sn
|
|||
Attempts: attemptsMade,
|
||||
Escalated: escalated,
|
||||
EscalationModel: escModel,
|
||||
BankFlags: last.bankFlags,
|
||||
}
|
||||
if disposition == DispOK {
|
||||
sr.Text = last.text
|
||||
|
|
@ -243,12 +264,18 @@ func (r *Runner) commitSanitizedExport(st config.Stage, ch Chunk, job *store.Job
|
|||
// stageAttempt is the result of one attempt (a checkpoint hit or a fresh call).
|
||||
type stageAttempt struct {
|
||||
reqHash string
|
||||
attempt int // the attempt index (for a derived checkpoint's Attempt column)
|
||||
cls classification
|
||||
// stripped is the resolved cosmetic-strip EXPORT text classifyOutput computed ONCE (T3.4):
|
||||
// non-empty only when cls.Reason == FlagSanitizerStripped. runStage ships it verbatim instead of
|
||||
// re-deriving stripCosmetic, so the "non-empty & clean" invariant classifyOutput asserts is the
|
||||
// exact text committed to the derived export checkpoint.
|
||||
stripped string
|
||||
// bankStripped is the banknote-CLEANED draft (WS4 point 8): non-empty only for a translator draft
|
||||
// that carried a ⟦TM-BANK-v1⟧ block, when the channel is enabled. runStage commits it as a derived
|
||||
// export checkpoint and re-points final_hash so the editor/export read the cleaned draft on resume.
|
||||
bankStripped string
|
||||
bankFlags bankFlags // per-chunk banknote telemetry (point 10) for the translator draft
|
||||
text string
|
||||
usage llm.Usage
|
||||
finish string
|
||||
|
|
@ -267,7 +294,7 @@ type stageAttempt struct {
|
|||
func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID string, ch Chunk, job *store.Job, attempt, maxTokens int, msgs []llm.Message, escalation, isFinal bool) (stageAttempt, error) {
|
||||
reqHash := RequestHash(r.Book.BookID, ch.Chapter, ch.ChunkIdx, attempt, st.Name, st.Role, model,
|
||||
st.Temperature, st.Reasoning, false, maxTokens, snapID, msgs)
|
||||
att := stageAttempt{reqHash: reqHash, modelActual: model}
|
||||
att := stageAttempt{reqHash: reqHash, attempt: attempt, modelActual: model}
|
||||
|
||||
// Resume on the attempt axis: a checkpoint means THIS attempt already happened
|
||||
// and was billed — classify its text and never re-bill (kill -9 loses ≤1 call;
|
||||
|
|
@ -282,11 +309,16 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID
|
|||
r.Log.DebugContext(ctx, "checkpoint usage_json unreadable; tokens report as zero", "hash", reqHash[:12], "err", uerr)
|
||||
}
|
||||
att.usage = usage
|
||||
att.text = cp.ResponseText
|
||||
att.finish = cp.FinishReason
|
||||
att.modelActual = cp.ModelActual
|
||||
att.cumCost = cp.CostUSD
|
||||
att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, cp.ResponseText, cp.FinishReason, isFinal)
|
||||
// Banknote: slice the block off the RAW checkpoint text BEFORE classify + before feeding the
|
||||
// editor (WS4 points 1-3,8). The checkpoint stores the RAW draft; resume re-derives the clean
|
||||
// text deterministically (a no-op returning the raw text for a channel-off / no-separator draft).
|
||||
cleanText, bankStripped, bankFlags := r.applyBanknote(st.Role, cp.ResponseText, cp.FinishReason)
|
||||
att.text = cleanText
|
||||
att.bankStripped, att.bankFlags = bankStripped, bankFlags
|
||||
att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, cleanText, cp.FinishReason, isFinal)
|
||||
rl := r.baseRequestLog(st, ch, model, reqHash)
|
||||
rl.ModelActual = cp.ModelActual
|
||||
rl.TMHit, rl.OK, rl.FinishReason, rl.Degraded = true, att.cls.ok(), cp.FinishReason, degradedTag(att.cls)
|
||||
|
|
@ -463,8 +495,13 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID
|
|||
// Classify AFTER the money is durably settled+checkpointed. F4 lives here: a
|
||||
// non-empty truncated length draft is now classified (flagged/retried), never
|
||||
// silently passed downstream as OK; an empty completion is flagged too, not a
|
||||
// run-crash.
|
||||
att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, resp.Text, resp.FinishReason, isFinal)
|
||||
// run-crash. The banknote block is sliced off the RAW draft FIRST (WS4 points 1-3,8): the
|
||||
// checkpoint above holds the raw response, but classify/coverage/echo and the editor see the
|
||||
// CLEANED translation, and the cleaned text is committed as a derived export in runStage.
|
||||
cleanText, bankStripped, bankFlags := r.applyBanknote(st.Role, resp.Text, resp.FinishReason)
|
||||
att.text = cleanText
|
||||
att.bankStripped, att.bankFlags = bankStripped, bankFlags
|
||||
att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, cleanText, resp.FinishReason, isFinal)
|
||||
|
||||
rl := r.baseRequestLog(st, ch, model, reqHash)
|
||||
rl.ModelActual = modelActual
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
42
backend/internal/pipeline/wave.go
Normal file
42
backend/internal/pipeline/wave.go
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
package pipeline
|
||||
|
||||
// wave.go: the W0 (deterministic, $0) pre-compute of the wave runner (WS1 §1б). This is the FIRST piece
|
||||
// of the wave executor built ahead of the driver switch (R1 residual): the sticky-chain memory selection
|
||||
// is a CROSS-CHUNK sequential dependency (a chunk's sticky_prev is the union of the prior chunks' exact
|
||||
// matches in the same chapter), so the parallel waves cannot recompute it per-chunk — it must be
|
||||
// precomputed once in W0. precomputeSticky replicates EXACTLY the sequential driver's inline computation
|
||||
// (bookrun.go's per-chapter reset + memory.Select + sticky advance), and the sequential driver now
|
||||
// CONSUMES it — so it is byte-identical by construction, and the golden test proves the injection bytes
|
||||
// did not move. When the driver switches to waves (residual), the waves consume this same precompute.
|
||||
//
|
||||
// Select is a PURE deterministic function of (chunk, chapter, sticky_prev, budget, frozen-bank) with NO
|
||||
// model output, so precomputing it reproduces byte-for-byte the injection the sequential runner rendered
|
||||
// (parallelism does not change the bytes — the order is fixed and the input carries no model state).
|
||||
|
||||
// precomputeSticky computes the per-chunk memory selection for EVERY chunk in one ordered W0 pass
|
||||
// (WS1 §1б): the sticky window is reset at each chapter boundary (a new chapter is a scene change) and
|
||||
// advanced by each chunk's exact matches, exactly as the sequential loop did inline. Returns a slice
|
||||
// aligned index-for-index with `chunks`. A nil bank (report path / no glossary) yields zero-valued
|
||||
// selections (no injection), so a caller can index it unconditionally.
|
||||
func precomputeSticky(chunks []Chunk, mem *MemoryBank, budget int) []memorySelection {
|
||||
out := make([]memorySelection, len(chunks))
|
||||
if mem == nil {
|
||||
return out
|
||||
}
|
||||
var stickyWin []map[string]injectionDisposition
|
||||
prevChapter := 0
|
||||
for i, ch := range chunks {
|
||||
if ch.Chapter != prevChapter {
|
||||
stickyWin = nil // chapter boundary: reset scene-inertia (A5)
|
||||
prevChapter = ch.Chapter
|
||||
}
|
||||
sel := mem.Select(ch.Text, ch.Chapter, unionSticky(stickyWin), budget)
|
||||
out[i] = sel
|
||||
// Advance the sticky window with THIS chunk's exact matches (keep the last stickyDepth).
|
||||
stickyWin = append(stickyWin, sel.activeIDs)
|
||||
if len(stickyWin) > stickyDepth {
|
||||
stickyWin = stickyWin[len(stickyWin)-stickyDepth:]
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
|
@ -79,6 +79,12 @@ type RetrievalState struct {
|
|||
// count is a loud seed-hygiene signal (a draft term nesting over an approved term to reconcile).
|
||||
NTrustGatedSuppress int
|
||||
TrustGateDetail string // JSON of the refused suppressor→protected pairs
|
||||
// NBanknoteLines / BanknoteParseFail / BanknoteTruncated are the banknote channel's per-chunk
|
||||
// telemetry (WS4 point 10): the accepted ⟦TM-BANK-v1⟧ line count and the parse-fail / truncation
|
||||
// flags (0/1) of the translator draft. Zero for every channel-off / non-banknote chunk.
|
||||
NBanknoteLines int
|
||||
BanknoteParseFail int
|
||||
BanknoteTruncated int
|
||||
}
|
||||
|
||||
// ReplaceGlossary replaces a book's ENTIRE glossary (rows + aliases) with the given
|
||||
|
|
@ -244,8 +250,9 @@ func (s *Store) UpsertRetrievalState(rs RetrievalState) error {
|
|||
book_id, chapter, chunk_idx, snapshot_id,
|
||||
n_exact_hits, n_sticky, n_ambiguous_flagged, n_spoiler_blocked, n_evicted,
|
||||
embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids,
|
||||
n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail, updated_at
|
||||
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,datetime('now'))
|
||||
n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail,
|
||||
n_banknote_lines, banknote_parse_fail, banknote_truncated, updated_at
|
||||
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,datetime('now'))
|
||||
ON CONFLICT (book_id, chapter, chunk_idx) DO UPDATE SET
|
||||
snapshot_id = excluded.snapshot_id,
|
||||
n_exact_hits = excluded.n_exact_hits,
|
||||
|
|
@ -261,11 +268,15 @@ func (s *Store) UpsertRetrievalState(rs RetrievalState) error {
|
|||
style_detail = excluded.style_detail,
|
||||
n_trust_gated_suppress = excluded.n_trust_gated_suppress,
|
||||
trust_gate_detail = excluded.trust_gate_detail,
|
||||
n_banknote_lines = excluded.n_banknote_lines,
|
||||
banknote_parse_fail = excluded.banknote_parse_fail,
|
||||
banknote_truncated = excluded.banknote_truncated,
|
||||
updated_at = excluded.updated_at`,
|
||||
rs.BookID, rs.Chapter, rs.ChunkIdx, rs.SnapshotID,
|
||||
rs.NExactHits, rs.NSticky, rs.NAmbiguousFlagged, rs.NSpoilerBlocked, rs.NEvicted,
|
||||
rs.EmbeddingTierUsed, rs.NPostcheckMiss, rs.PostcheckDetail, rs.InjectedIDs,
|
||||
rs.NStyleFlags, rs.StyleDetail, rs.NTrustGatedSuppress, rs.TrustGateDetail)
|
||||
rs.NStyleFlags, rs.StyleDetail, rs.NTrustGatedSuppress, rs.TrustGateDetail,
|
||||
rs.NBanknoteLines, rs.BanknoteParseFail, rs.BanknoteTruncated)
|
||||
return err
|
||||
}
|
||||
|
||||
|
|
@ -277,12 +288,14 @@ func (s *Store) GetRetrievalState(bookID string, chapter, chunkIdx int) (*Retrie
|
|||
err := s.r.QueryRowContext(ctx, `
|
||||
SELECT snapshot_id, n_exact_hits, n_sticky, n_ambiguous_flagged, n_spoiler_blocked,
|
||||
n_evicted, embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids,
|
||||
n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail
|
||||
n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail,
|
||||
n_banknote_lines, banknote_parse_fail, banknote_truncated
|
||||
FROM retrieval_state WHERE book_id = ? AND chapter = ? AND chunk_idx = ?`,
|
||||
bookID, chapter, chunkIdx).Scan(
|
||||
&rs.SnapshotID, &rs.NExactHits, &rs.NSticky, &rs.NAmbiguousFlagged, &rs.NSpoilerBlocked,
|
||||
&rs.NEvicted, &rs.EmbeddingTierUsed, &rs.NPostcheckMiss, &rs.PostcheckDetail, &rs.InjectedIDs,
|
||||
&rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail)
|
||||
&rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail,
|
||||
&rs.NBanknoteLines, &rs.BanknoteParseFail, &rs.BanknoteTruncated)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
|
|
@ -298,7 +311,8 @@ func (s *Store) RetrievalStatesForBook(bookID string) ([]RetrievalState, error)
|
|||
return queryAll(s.r, `
|
||||
SELECT chapter, chunk_idx, snapshot_id, n_exact_hits, n_sticky, n_ambiguous_flagged,
|
||||
n_spoiler_blocked, n_evicted, embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids,
|
||||
n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail
|
||||
n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail,
|
||||
n_banknote_lines, banknote_parse_fail, banknote_truncated
|
||||
FROM retrieval_state WHERE book_id = ?
|
||||
ORDER BY chapter, chunk_idx`,
|
||||
func(rows *sql.Rows) (RetrievalState, error) {
|
||||
|
|
@ -306,7 +320,8 @@ func (s *Store) RetrievalStatesForBook(bookID string) ([]RetrievalState, error)
|
|||
err := rows.Scan(&rs.Chapter, &rs.ChunkIdx, &rs.SnapshotID, &rs.NExactHits, &rs.NSticky,
|
||||
&rs.NAmbiguousFlagged, &rs.NSpoilerBlocked, &rs.NEvicted, &rs.EmbeddingTierUsed,
|
||||
&rs.NPostcheckMiss, &rs.PostcheckDetail, &rs.InjectedIDs,
|
||||
&rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail)
|
||||
&rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail,
|
||||
&rs.NBanknoteLines, &rs.BanknoteParseFail, &rs.BanknoteTruncated)
|
||||
return rs, err
|
||||
}, bookID)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -282,6 +282,16 @@ var migrations = []string{
|
|||
ALTER TABLE retrieval_state ADD COLUMN n_trust_gated_suppress INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE retrieval_state ADD COLUMN trust_gate_detail TEXT NOT NULL DEFAULT '';
|
||||
`,
|
||||
// v9 (WS4): the banknote channel's per-chunk telemetry (integration point 10) — the accepted line
|
||||
// count and the parse-fail / truncation flags of the translator draft's ⟦TM-BANK-v1⟧ block, riding
|
||||
// the same per-chunk observability row as the glossary post-check and the cheap style flaggers.
|
||||
// Recomputed deterministically each run (self-heals on resume), NOT wire-load-bearing (observability,
|
||||
// never a disposition). Zero for every channel-off / non-banknote chunk (the common case).
|
||||
`
|
||||
ALTER TABLE retrieval_state ADD COLUMN n_banknote_lines INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE retrieval_state ADD COLUMN banknote_parse_fail INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE retrieval_state ADD COLUMN banknote_truncated INTEGER NOT NULL DEFAULT 0;
|
||||
`,
|
||||
}
|
||||
|
||||
// DESIGN NOTE (D21.10 — reserved memory-bank v2 record types; Ф2, NOT a migration, NOT code).
|
||||
|
|
|
|||
|
|
@ -1,7 +1,19 @@
|
|||
> **⚠ АКТУАЛИЗИРОВАН ПОД ПРОДОЛЖЕНИЕ (19.07, пост-D39.13; исходная версия — в git-истории).** Сессия №1 сдала
|
||||
> Block A + ядра — ПРИНЯТО И ЗАЛЕНДЕНО (`85f5b9e`): WS2 ✅ · WS1a/b/c-инфра ✅ · WS4-core ✅ · WS6-конфиг-тай ✅.
|
||||
> **Эта версия — стартовая для СЕССИИ-ПРОДОЛЖЕНИЯ (резидуал R1–R5).** Сданное — под тестами, НЕ пере-строить;
|
||||
> дизайны резидуала — `backend/PACK11_REPORT.md` §Резидуал (авторитет наравне с планом; исполнять, не пере-изобретать).
|
||||
> **⚠ АКТУАЛИЗИРОВАН ПОД R1-ПРОДОЛЖЕНИЕ (19.07, пост-D39.14; предыдущие версии — в git-истории).** Сессия №1
|
||||
> сдала Block A (`85f5b9e`, D39.13); сессия №2 сдала R2–R5 + R1-аддитив (D39.14). **Осталось: R1 драйвер-свитч +
|
||||
> 3-пунктовый фикс-лист рубежа-2 (ниже).** Дизайны — `backend/PACK11_CONT_REPORT.md §R1` + `PACK11_REPORT.md §Резидуал`
|
||||
> (авторитет наравне с планом). Сданное — под тестами, НЕ пере-строить.
|
||||
>
|
||||
> **ФИКС-ЛИСТ рубежа-2 (D39.14; все латентны — банкнота off по дефолту, эмиссия за R1 — лендинг не блокировали;
|
||||
> закрыть В ЭТОЙ сессии, они трогают ровно R1-файлы):**
|
||||
> - **FL-1 (MINOR):** свернуть `bankMaxLines`/`bankTokenBudget` в `banknoteSnap` (`snapshot.go`) — сейчас
|
||||
> `max_tokens += bankTokenBudget` wire-affecting (∈RequestHash), но не свёрнут → правка конста без бампа
|
||||
> `bankParserVersion` = тихий re-bill книги на резюме (loudness дисциплиной, не механизмом).
|
||||
> - **FL-2 (MINOR, observability):** `resumeFromChunkStatus` (`resume.go:22`) не ставит `BankFlags` → банкнота-
|
||||
> телеметрия (`n_banknote_lines/parse_fail/truncated`) обнуляется на резюме готовых чанков. Re-derive из сырого
|
||||
> чекпоинта, ИЛИ не обнулять эти колонки на резюме, ИЛИ поправить ложный коммент «self-heals on resume».
|
||||
> - **FL-3 (PARTIAL, порт-fidelity):** `miner_emit.go:68` идёт по полному `mr.ranked`; референс капит `a3[:200]`/
|
||||
> `[:30]` ДО фильтров (`eval/exp16/{alias.py:165, emit_owner_sheets.py:143,162,169}`) → карта подписи шумнее
|
||||
> референса. Добавить top-N кап перед emission-фильтрами. **Закрыть ДО живого W1.5** (владелец подписывает карту).
|
||||
|
||||
# Промт: сессия БЭКЕНД — единый пак по плану 11 (волновой раннер · чанкер/единицы · банк W1.5 · банкнота · чекеры · армы), 2026-07-19
|
||||
|
||||
|
|
|
|||
|
|
@ -9,6 +9,10 @@
|
|||
> - **Ждём от владельца:** запуск exp15 (бюджет ≤$15 подтверждён) · research/20 §E: мини-голд алиасов (~20–30 мин) / бюджет ≤$5 полигон-стадии / политика канона / жанр-паки / реплика ja→ru · чтение пере-прогона (после очереди) · старые висящие: планка запуска (лучше-фана/гибрид/издательский) · билингв-якорь пилота (D25.9-Q1) · контаминация пилот-корпуса (D27.4) · publishable/waiver (D25.1) · FN-bound L3 (D25.4) · юр-пакет · провенанс 12-*-доков.
|
||||
> - Архивы хроники: `archive/PROGRESS-2026-07-04-10.md` (D31) · `archive/PROGRESS-2026-07-10-13.md` (D39.6-гигиена). Записи ниже — живая эра D39.
|
||||
|
||||
## Оркестратор №7 — пак-11 continuation принят и залендён (D39.14), рубеж-2-долг погашен, 19.07 (четвёртая фаза)
|
||||
|
||||
Continuation-сессия сдала R2–R5 + R1-аддитив (санкц. порядок вариант-1). **Залендено:** R2 Go-майнер (паритет EXACT: 13618, 古月:22, recall 0.9649) · R3 банкнота 12-точек (на посл. раннере, переезжают в волны как есть) · R4 DC1/DC2/DC6+DC3-gender+`export --pairs` (2 golden re-capture masked-чист) · R5 2 арм-конфига · R1-аддитив (snapshotIDForWave + W0-precompute байт-идентично). R1 драйвер-свитч (W1/W1.5/W2 + структурный golden-рерайт) — санкц. резидуал. **Приёмка исполнением:** build/vet/race сам, ~40 пинов свежим `-count=1` (майнер-паритет TM_MINER_PARITY EXACT), независимая masked-golden репликация (0 изменений final_text/disposition), построчное чтение деньги-пути (settle пишет raw, срез после durable settle — деньги нетронуты). **Долг рубежа-2 (D39.13) ПОГАШЕН:** 8-агентный адверсариал по обоим кускам, 5 линз refute-by-default, $0 — **3 находки (все MINOR/PARTIAL, 0 refuted); все CRITICAL/MAJOR-линзы пусты** (деньги/wire/порт/резюм/base-enriched/residual — clean bill). Фикс-лист FL-1 (bankTokenBudget не свёрнут в banknoteSnap) · FL-2 (banknote-телеметрия обнуляется на резюме, observability) · FL-3 (miner emission без top-N капа референса) — все латентны (банкнота off по дефолту, эмиссия за R1), лендинг не блокируют → R1-продолжение (промт активен, фикс-лист вписан). §0.1-заметка владельцу: DC-данные = код-консты за isRuTarget-гейтом (сеам защищает; config-extraction — открытый вопрос, увязать с ja→ru). Очередь: R1-продолжение → приёмка → resnapshot → пере-прогон → чтение.
|
||||
|
||||
## Оркестратор №7 — пак-11 Block A принят и залендён (D39.13), 19.07 (третья фаза сессии)
|
||||
|
||||
Бэкенд-сессия сдала Block A + ядра с ЧЕСТНЫМ резидуалом (мандат «резидуал > сломанный суит»): WS2 целиком (output-бюджет чанкера, edit-единицы grouping-of-whole-chunks [девиация ратифицирована: 37 единиц / 0 straddle воспроизведены, гарантия W2-реконструкции по построению], src→dst-блок редактора = закрытие D30.1, `render-format-version`, golden re-capture) · WS1a eager-clients (гонка закрыта) · WS1b rate-guard (`ratelimit.go`) · WS1c base/enriched `MemoryVersion`-split (enriched байт-идентичен — снапшот не двинулся) · WS4-core банкнота-парсер (байт-верный порт). **Приёмка — верификация исполнением:** build/vet/race сам, 27 пинов свежим `-count=1` вербозно, независимая masked-golden репликация (все категории объяснимы), построчное чтение несущих диффов. Сессионный 9-агентный чекпойнт: 3 confirmed пофикшены / 3 refuted. **Долг: мульти-агентный рубеж-2 приёмки не гонялся (лимит-режим) — гасится при приёмке продолжения (прецедент D39.2).** Резидуал R1–R5 с точными дизайнами — `backend/PACK11_REPORT.md`; промт пака АКТИВЕН со статус-баннером. Записка продолжению-сессии: стартовать по промту + отчёту, сданное не пере-строить.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
Loading…
Add table
Reference in a new issue