Land backend pack-11 block A: output-token chunker with edit units, src-to-dst editor block, eager clients, per-model rate guard, memory version split, banknote parser core
This commit is contained in:
parent
1f285250ab
commit
85f5b9e633
27 changed files with 1402 additions and 303 deletions
239
backend/PACK11_REPORT.md
Normal file
239
backend/PACK11_REPORT.md
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
# PACK11_REPORT — бэкенд-сессия по плану 11 (2026-07-19)
|
||||
|
||||
> Строил по `docs/architecture/11-implementation-plan.md` (ратифицирован D39.12). **НЕ коммитил** —
|
||||
> сдача на приёмку оркестратору. Верификация исполнением: `go build ./... && go vet ./... &&
|
||||
> go test -race ./...` — зелёное (логи ниже). Все несущие изменения сверены с живым HEAD-кодом.
|
||||
|
||||
## 0. Честный статус-обзор (что сдано / что резидуал)
|
||||
|
||||
Пак 11 — это ФАКТИЧЕСКИ четыре пакета работ (§7 плана: пак-2 слой-1 + пак-3 слой-4 + пак-4 слой-5 +
|
||||
конфиг-пак). За одну сессию доведено до **зелёного, оттестированного, лендабельного** состояния:
|
||||
|
||||
| WS | Компонент | Статус |
|
||||
|---|---|---|
|
||||
| **WS2** | чанкер output-бюджет + edit-единицы + oversized-флаг + src→dst редактор + снапшот-фолды + golden re-capture | ✅ **СДАНО** (тесты + golden) |
|
||||
| **WS1a** | eager-build клиентов в W0 + read-only `client()` (гонка `runner.go` закрыта) | ✅ **СДАНО** (`-race`) |
|
||||
| **WS1b** | пер-модельный rate-guard (семафор конкуренции + пейсинг) | ✅ **СДАНО** (concurrency-тест) |
|
||||
| **WS1c-инфра** | base/enriched `MemoryVersion`-split (механика пер-волнового снапшота, «переоплата ОДНА») | ✅ **СДАНО** (unit-тест) |
|
||||
| **WS4-core** | банкнота: pure `splitBanknote`/`parseBanknote` (`⟦TM-BANK-v1⟧`, толерантный сплит, Han-в-src, finish-truncation) | ✅ **СДАНО** (7 инвариант-тестов; Go↔Python паритет к `banknote.py`, пин 95/65/0/0 воспроизведён рефересом) |
|
||||
| **WS6** | mistral rate_limit конфиг (свап-арм за rate-guard); арм-механика (editor-model = конфиг → снапшот per-арм) уже в коде | ✅ **частично** (конфиг-тай; арм-пайплайн-конфиги — резидуал) |
|
||||
| **WS1c-executor** | параллельный волновой ИСПОЛНИТЕЛЬ W1/W1.5/W2 + пер-волновой снапшот-wiring + golden-рерайт + resume-across-waves + kill-9 N-резерв | ⚠️ **РЕЗИДУАЛ** (дизайн ниже, §Резидуал) |
|
||||
| **WS4-integration** | 12 точек интеграции банкноты (срез ДО classify, derived-checkpoint, snapshot-фолд, телеметрия) | ⚠️ **РЕЗИДУАЛ** (ядро+дизайн готовы) |
|
||||
| **WS3** | Go-майнер V-A→V-C + алиас + консолидация + сид-дельта + `tmctl seed-lint` + `Source:mined` | ⚠️ **РЕЗИДУАЛ** (Source-split инфра готова; порт — резидуал) |
|
||||
| **WS5** | DC-чекеры + omission-бэкстоп + данные пакета пары + `tmctl export --pairs` | ⚠️ **РЕЗИДУАЛ** (дизайн ниже) |
|
||||
|
||||
**Почему не весь объём:** волновой ИСПОЛНИТЕЛЬ (WS1c) — это ре-архитектура модели исполнения
|
||||
(последовательный per-chunk-all-stages → W1 draft-∥ / W1.5-стоп / W2 edit-∥-over-units), которая
|
||||
инвалидирует golden + ~10 последовательно-модельных тестов (runner_memory/export/status/cheapgates/
|
||||
quality/escalation_budget/maxtokens_floor) и добавляет двух-фазный CLI-флоу. Довести это до зелёного
|
||||
за сессию БЕЗ поломки суита нереалистично. Решение (мандат «честный резидуал > фейк-комплит, не
|
||||
сдавать сломанное»): сдать **несущую БЕЗОПАСНУЮ инфраструктуру волн** (eager-clients, rate-guard,
|
||||
base/enriched split, edit-units) как лендабельную и зелёную, а параллельный исполнитель —
|
||||
резидуалом с точным дизайном (§Резидуал). Ни один сданный кусок не оставляет суит красным.
|
||||
|
||||
---
|
||||
|
||||
## 1. WS2 — чанкер-бюджет + edit-единица (план §2) — ✅ СДАНО
|
||||
|
||||
Per-буллет чек-лист (буллеты промта):
|
||||
|
||||
- ✅ **Бюджет draft-чанков в ВЫХОДНЫХ токенах** `est_out = 1.1978·cjk + 0.3852·other`, конфиг
|
||||
`config.Segmentation` (per-pair, снапшот-folded через `segmentationSnap`). Классы символов —
|
||||
**РЕАЛЬНЫЕ `unicode.RangeTable`** (`Han|Hiragana|Katakana|Hangul`, `tokenClassCounts` = те же, что
|
||||
`EstimateTokens`), НЕ ord-диапазоны (§0.1 общность). Пин `target_out=1797 → 56 чанков` —
|
||||
воспроизведён на пере-прогонном корпусе (см. §Верификация ниже).
|
||||
- ✅ **Edit-единица = глава; дефолт `EditCeilingOut=3200`.** Реализовано как **greedy-группировка
|
||||
ЦЕЛЫХ draft-чанков до потолка** (`assignEditUnits`). **ДЕВИАЦИЯ-с-обоснованием (эскалация-нота):**
|
||||
план §2(б) описывает edit-единицы как paragraph-greedy-pack; я реализовал grouping-of-whole-chunks
|
||||
потому что независимые нарезки (para-pack draft @1797 и para-pack units @3200) МОГУТ дать
|
||||
straddle draft-чанка через границу единицы → сломанная реконструкция draft для W2. **Эмпирически
|
||||
проверено на реальном 25-гл корпусе:** grouping-whole-chunks даёт ТЕ ЖЕ **37 единиц**, что
|
||||
para-pack, при **0 straddle** (скрипт в scratchpad; вывод: `{draft:56, para_units:37,
|
||||
grouped_units:37, straddle:0}`). Т.е. на этом корпусе оба определения СОВПАДАЮТ, а grouping —
|
||||
робастный супермножество-инвариант (гарантирует чистую реконструкцию всегда). Пин 37 сохранён.
|
||||
- ✅ **8000-арм НЕ открыт** — дефолт 3200; `edit_ceiling_out` остаётся config-tunable (потолок —
|
||||
конфиг, арм гейтнут §11), валидация `EditCeilingOut ≥ DraftBudgetOut`.
|
||||
- ✅ **`Chunk` аддитивно:** `EstOut float64`, `OversizedSentence bool`, `EditUnitID int`. Контракт
|
||||
`Chapter/ChunkIdx/Text` цел (тесты `TestSplitChunks*` + fuzz lossless-tiling зелёные).
|
||||
- ✅ **Мега-предложение** = passthrough + `OversizedSentence=true` (БЕЗ клауза-сплита). Пин
|
||||
`TestSplitChunksOversizeSentenceIsOwnChunk`.
|
||||
- ✅ **Снос мёртвых `STMDepth`/`OverlapTokens`** из `config.ContextAssembly` + `contextSnap` +
|
||||
yaml-конфигов (carryover не строим). Структурная смена `contextSnap` → строка манифеста §8.
|
||||
- ✅ **src→dst-формат `renderEditorConstraintBlock`** (`- 源термин → «dst»`), CONFIRMED-only ЦЕЛА,
|
||||
дедуп по `(src,dst)` (омоним-widening). Обоснование — билингв-редактор + омонимичный dst (§2в
|
||||
ревью-1 F4). **Выделенный `render-format-version`** снапшот-компонент (НЕ `memoryMatchVersion`).
|
||||
Пины `TestRenderEditorConstraintBlock` + `TestRunnerMemoryInjectionAndPostcheck` обновлены.
|
||||
- ✅ **Сайзинг max_tokens редактора** — НЕ трогал (F8: существующий `stagerun.go:87-90` сайзит от
|
||||
черновика `prev`; новой формулы не вводил — как и предписано).
|
||||
- ⚠️ **Инъекция W2-единицы (СВЕЖИЙ Select над enriched-банком)** — часть волнового ИСПОЛНИТЕЛЯ
|
||||
(резидуал WS1c). Механика src→dst-блока и enriched-версии готова; вызов свежего Select по тексту
|
||||
единицы делается в W2-проходе (резидуал). **Whole-chapter Select-бюджет (§2е открытый)** — решать
|
||||
в исполнителе (резидуал-нота).
|
||||
|
||||
Снапшот-фолды WS2 (§8): `chunkerVersion` v4→v5, `segmentationSnap`, `render_format_version`, снос
|
||||
`stm_depth`/`overlap_tokens`. Golden re-captured — masked-дифф ниже (только versions/new-fields/
|
||||
editor-src2dst; НИ одной несвязанной content-правки; фикстура НЕ ре-чанкается — 20 чанк-строк до/после).
|
||||
|
||||
## 2. WS1 — волновой раннер (план §1)
|
||||
|
||||
- ✅ **WS1a: Eager-build ВСЕХ клиентов в W0** (`buildClients` над `reachableModels()` =
|
||||
`Stages[].Model ∪ EscalateTo`), вызван в `TranslateBook` W0-прелюдии ДО любой горутины. `client()`
|
||||
теперь **READ-ONLY** — громкий отказ на miss (закрывает гонку `runner.go:171-181` без лока;
|
||||
трипваер на будущую 3-ю модель-ось). `BuildClient` не требует ключа → безопасно на key-less resume.
|
||||
`-race` чист.
|
||||
- ✅ **WS1b: Пер-модельный rate-guard** (`ratelimit.go`): семафор конкуренции `models.yaml
|
||||
rate_limit.max_concurrency` + опц. пейсинг `min_interval_ms`. **Транспорт-ось, НЕ снапшот** (wire
|
||||
не трогает). Построен в W0 (`buildRateGuards`), read-only в волнах; обёрнут вокруг ОДНОГО
|
||||
`client.Complete` в `runAttempt` (ctx-aware acquire/release, reservation освобождается при
|
||||
ctx-cancel). Пины: `TestRateGuardLimitsConcurrency` (семафор РЕАЛЬНО ограничивает — cap=2/20
|
||||
воркеров, observed-max ≤ 2), nil/unlimited no-op, ctx-cancel без leak, pacing. **`waves.workers`
|
||||
конфиг** — вводится с исполнителем (резидуал).
|
||||
- ✅ **WS1c-инфра: base/enriched `MemoryVersion`-split.** `MemoryBank.BaseVersion()` = хеш approved
|
||||
`Source∈{seed,ruby,auto}` (БЕЗ mined); `Version()` (enriched) = все approved вкл. `Source:mined`.
|
||||
`computeMemoryVersionScoped(excludeMined)`; enriched-путь **байт-идентичен** прежнему
|
||||
`computeMemoryVersion` (домен-сепаратор только в base) → текущий снапшот не двигается от split-а.
|
||||
Пин `TestBaseEnrichedMemoryVersionSplit`: добавление mined-approved строки двигает ТОЛЬКО enriched,
|
||||
base стабилен (= «переоплата ОДНА» механизм); base≠enriched (домен-разделены). **`Source:mined`**
|
||||
как значение enum — аддитивно (колонка `source` — свободный TEXT, миграция store не нужна;
|
||||
подтверждено чтением `migrate.go`).
|
||||
- ⚠️ **WS1c-executor (РЕЗИДУАЛ):** пер-волновой снапшот-wiring (`snapshot_W1`/`snapshot_W2` upsert),
|
||||
W0 sticky-precompute, W1/W2 параллельный dispatch, W1.5-стоп + mined-write-путь, wave-sequencing
|
||||
инвариант (W2 не пере-рендерит draft), money-под-∥ kill-9 N-резерв, scheduler-aware окна. Дизайн —
|
||||
§Резидуал.
|
||||
|
||||
## 3. WS4-core — банкнота-парсер (план §4б) — ✅ СДАНО
|
||||
|
||||
`internal/pipeline/banknote.go` — байт-верный порт `exp16/banknote.py`:
|
||||
- `splitBanknote(output)` → срез блока `⟦TM-BANK-v1⟧` (clean = rstrip до сепаратора, block = strip("\n"));
|
||||
- `parseBanknote(block, truncated)` → толерантный сплит `\t| {2,}|\s*\|\s*`, тип∈{name,place,title,term,
|
||||
nickname}+дефолт term, src ДОЛЖЕН нести Han `[㐀-鿿]` (точный диапазон U+3400–U+9FFF), truncation-толерантность
|
||||
на ПОСЛЕДНЕЙ строке (флаг `Truncated`, не parse_fail);
|
||||
- `bankParserVersion="banknote-v1"` (для будущего `banknoteSnap{enabled,parser_version}`).
|
||||
- Тесты (`banknote_test.go`): no-SEP→всё clean; срез блока (утечки нет); толерантный сплит tab/≥2проб/pipe;
|
||||
non-Han src=bad; truncation True→флаг/False→parse_fail; пустой блок; SEP≠note-слово. **Все PASS.**
|
||||
- **Паритет:** референс `ws4_banknote_verify.py` воспроизводит пин **95 строк / 65 distinct / 0 fail / 0
|
||||
trunc** + truncation-поведение + инварианты — идентично моему Go-порту (сверено исполнением).
|
||||
|
||||
**WS4-integration (12 точек) — резидуал** (§Резидуал R3): требует правок `chunkrun.go`/`stagerun.go`/
|
||||
`disposition.go`/`snapshot.go` (файлы под адверсариал-ревью этой сессии — не трогал во избежание дрейфа
|
||||
цитат; ядро + точный дизайн готовы).
|
||||
|
||||
## 4–5. WS3 / WS5 — ⚠️ РЕЗИДУАЛ (дизайн в §Резидуал)
|
||||
|
||||
Не начаты в коде (кроме `Source:mined`-инфра из WS1c, которая — предусловие WS3-mined-write). Причина
|
||||
— бюджет сессии ушёл в несущий каркас Block A + верификацию + WS4-ядро. Дизайны/пред-условия — §Резидуал.
|
||||
|
||||
---
|
||||
|
||||
## Багфиксы вне плана
|
||||
Пока нет (изменения строго по плану). Правки по ревью — см. §Адверсариал-ревью ниже (2 doc-фикса + 1 gofmt).
|
||||
|
||||
## Структурные переносы
|
||||
- Новый файл `internal/pipeline/ratelimit.go` (rate-guard) + `banknote.go` (WS4-ядро) — новые оси, не рефактор.
|
||||
- `SegBudget` (resolved segmentation) живёт в `chunker.go`; резолвер `segBudget()` — в `runner.go`
|
||||
(segmenter остаётся config-free). `renderFormatVersion` const — в `memory.go` рядом с рендерерами.
|
||||
|
||||
---
|
||||
|
||||
## Верификация (исполнением — логи, не слова)
|
||||
|
||||
```
|
||||
$ go build ./... → OK
|
||||
$ go vet ./... → OK (чисто)
|
||||
$ go test -race ./... → ВСЁ ЗЕЛЁНОЕ:
|
||||
ok cmd/tmctl 1.0s · config · ledger · llm · pipeline 19.4s · store
|
||||
$ gofmt -l <мои файлы> → пусто (все чисты; репо несёт ПРЕ-существующий gofmt-долг в ~13
|
||||
не-моих файлах под go1.26.4 — их не трогал)
|
||||
```
|
||||
|
||||
**Фикстуры плана (исполнены рефересами `eval/design11/*` под `eval/.venv`):**
|
||||
- WS2 фертильность/нарезка: `56 чанков @1797` / `37 edit-единиц @3200` / **`0 straddle`** —
|
||||
подтверждено `ws2_straddle_check.py` (grouping-whole-chunks ≡ para-pack на корпусе).
|
||||
- WS4 банкнота: `95 строк / 65 distinct / 0 fail / 0 trunc` (`ws4_banknote_verify.py`) — мой Go-парсер
|
||||
воспроизводит контракт (7 инвариант-тестов PASS, идентичны демо `banknote.py`).
|
||||
|
||||
**Go-тесты, добавленные/обновлённые этой сессией (все PASS под `-race`):**
|
||||
`TestSplitChunks*` (+ EstOut/OversizedSentence/EditUnitID пины + `TestSplitChunksEditUnitGroupsWholeChunks`),
|
||||
`TestRateGuard*` (concurrency-cap реально ограничивает, ctx-cancel без leak, pacing, nil no-op),
|
||||
`TestBaseEnrichedMemoryVersionSplit`, `TestRenderEditorConstraintBlock` (src→dst), `TestRunnerMemoryInjectionAndPostcheck`,
|
||||
`TestSplitBanknote*`/`TestParseBanknote*`, `TestGoldenDeterminism` (re-captured).
|
||||
|
||||
## Адверсариал-ревью Block A (мандат: author≠reviewer, refute-by-default, 3 линзы) — ИСПОЛНЕН
|
||||
9-агентный воркфлоу (3 ревью-линзы: алгоритм-vs-план / сеньор-код-ревью / функциональное, каждая
|
||||
находка адверсариально верифицирована). **6 сырых находок → 3 подтверждено, 3 отвергнуто** (верификатор
|
||||
refuted как not-a-bug/hypothetical):
|
||||
- **CONFIRMED-1/2 (minor, doc):** комменты base/enriched говорили «equal until mined», но код
|
||||
ДОМЕН-РАЗДЕЛЯЕТ их (base≠enriched всегда). Код/дизайн ВЕРНЫ; комменты — нет. **ПОФИКШЕНО**
|
||||
(`memory.go:135`,`:179`).
|
||||
- **CONFIRMED-3 (nit, style):** `subChunk` struct не gofmt-выровнен. **ПОФИКШЕНО** (`gofmt -w`).
|
||||
- **REFUTED (верифицировано как не-баг):** (a) «edit-unit grouping диверджит от плана» → НЕ баг:
|
||||
grouping-whole-chunks ТРЕБУЕТСЯ для W2-реконструкции, ≤3200 by construction, а т.к.
|
||||
`chapterDraftChunks` НИКОГДА не режет абзац — каждая draft-граница = абзац-граница → удовлетворяет
|
||||
И §2б И §1б (сильнее, чем я аргументировал); 37 — описательная статистика, не решающая ветка кода.
|
||||
(b) rate-guard money-путь «нет интеграц-теста» → гипотетический будущий регресс, не дефект сданного
|
||||
кода (release-on-cancel корректен). (c) job-status «running» до acquire → корректная семантика,
|
||||
лог `calling model` маркирует queued-vs-wire. Диспозиции — в отчёте, тихих девиаций нет.
|
||||
|
||||
## Golden re-capture (WS2) — маскированный структурный дифф
|
||||
18 изменённых masked-строк, ВСЕ объяснимы: `chunker_version` v4→v5; снятые `stm_depth`/`overlap_tokens`;
|
||||
новое `segmentation:{1797,3200,1.1978,0.3852}`; новое `render_format_version:renderfmt-v1-editor-src2dst`;
|
||||
редактор-блок `- «dst»` → `- src → «dst»` (7 editor-body строк). Фикстура НЕ ре-чанкается. Полный
|
||||
masked-дифф — scratchpad `ws2_golden_masked_diff.txt`.
|
||||
|
||||
---
|
||||
|
||||
## РЕЗИДУАЛ — точные дизайны для следующей сессии
|
||||
|
||||
### R1. Волновой ИСПОЛНИТЕЛЬ (WS1c-executor) — несущий, самый крупный
|
||||
Каркас готов (eager-clients, rate-guard, base/enriched split, edit-units на чанках). Осталось:
|
||||
1. `snapshotIDForWave(wave)` — рефактор `snapshot.go`: fold ТОЛЬКО стадии волны (draft=translator-роль
|
||||
→ W1; editor-роль → W2) + версию банка волны (`BaseVersion` для W1, `Version` для W2). Апсертить
|
||||
ДВА снапшота.
|
||||
2. Партиция стадий по роли; edit-единица addressable как `(chapter, firstChunkIdxOfUnit, "edit")` в
|
||||
`chunk_status` (без миграции — units per-chapter, group-of-whole-chunks).
|
||||
3. W0 sticky-precompute: реплицировать последовательную sticky-цепочку ($0, `Select` src-производна)
|
||||
→ инъекц-байты draft ≡ последовательным (golden-пин).
|
||||
4. W1 dispatch: N воркеров тянут draft-чанки, `runStage(draft)` под `snapshot_W1`. W2 dispatch: N
|
||||
воркеров тянут edit-единицы, `prev` = конкатенация draft-выходов чанков единицы (из W1-чекпоинтов
|
||||
по `final_hash`), СВЕЖИЙ `Select` по тексту ВСЕЙ единицы над enriched-банком, `runStage(edit)` под
|
||||
`snapshot_W2`. **W2 НИКОГДА не пере-рендерит draft** (несущий инвариант §1в).
|
||||
5. W1.5: при пустой mined-дельте (WS3 не построен) — авто-продолжение; иначе стоп + карта подписи +
|
||||
mined-write-путь (штамп `Source:mined`, зеркало `rubyToCandidates`, НЕ через `loadGlossarySeed`).
|
||||
6. Money-под-∥: воркеры-горутины через single-writer store (уже сериализует). kill-9 тест с N живыми
|
||||
резервами (порт `store/kill9_test.go` на N). rate-guard уже интегрирован.
|
||||
7. Golden-рерайт под волновую модель + обновление ~10 последовательно-модельных тестов.
|
||||
|
||||
### R2. WS3 майнер — референс `eval/exp16/{arms,alias,canon,patterns,spread,palladius,banknote}.py`
|
||||
Новый пакет `internal/pipeline/miner*.go`. Порт V-A→V-C по ТОЧНЫМ порогам §3(б); дефолт-лемматизатор
|
||||
**B** (дроп Палладий-подканала → `古月:21→22`, скрин PASS). Инвариантные гарантии (EXACT): 13618-канд-СЕТ,
|
||||
recall 0.9649/0.9318, катастроф-скрин `方源/蛊/蛊师` ранги 0/1/2 ∈top-50. Реюз `memnorm.normalizeSourceKey`.
|
||||
`tmctl seed-lint` (сухой прогон фейл-лаудов `loadGlossarySeed`). Go↔Python паритет (Python=референс).
|
||||
`Source:mined`-инфра (base/enriched split) — **готова** (WS1c).
|
||||
|
||||
### R3. WS4 банкнота — референс `eval/exp16/banknote.py`, спека research/20 §B3
|
||||
Pure `split_banknote`/`parse_banknote` (`⟦TM-BANK-v1⟧`, ≤12 строк src\tdst\ttype, толерантный сплит,
|
||||
Han-в-src, finish=stop-only гейт). 12 точек: срез ДО `classifyOutput` (`chunkrun.go`); derived-checkpoint
|
||||
`sha256("tm-banknote-v1\x00"+reqHash+"\x00"+stripped)` через `PutDerivedCheckpoint` (прецедент
|
||||
`commitSanitizedExport` — **уже есть**); re-point `final_hash` OK-пути draft на derived ПОСЛЕ резолва
|
||||
эскалации (`last.stripped`); `banknoteSnap{enabled,parser_version}` в `snapshotID()` (фолд при enabled,
|
||||
зеркало `sanitizerSnap`); инструкция сноски В ФАЙЛ промпта; телеметрия `n_banknote_lines`/`parse_fail`/
|
||||
`truncated`. Пин: пере-парс 95 строк = 0 fail (`ws4_banknote_verify.py`).
|
||||
|
||||
### R4. WS5 DC-чекеры + omission-бэкстоп + данные пакета пары + `tmctl export --pairs`
|
||||
Расширить `cheapgates.go` DC1/2/6/7 (детерм. флаггеры, target-gated, наблюдаемость не гейт); DC3
|
||||
gender-инъекция (male/female/hidden-мандат) из seed-поля `Gender` в `renderEditorConstraintBlock`
|
||||
(фолд через `render-format-version` + `MemoryVersion`); omission-бэкстоп (substring по
|
||||
`{dst,decl.forms,aliases}` + лёгкий Go-стеммер, БЕЗ pymorphy3); данные пакета zh-ru (версионируемый
|
||||
снапшот-folded пакет); `tmctl export --pairs` (source-join уже в `Export`) — пред-условие §5(д)-FP.
|
||||
DC5-детекцию НЕ строить (74%/42% FP); hidden-ЧЕКЕР НЕ строить (Ф2-кореф). Референс `ws5_checkers_verify.py`.
|
||||
|
||||
### R5. WS6 арм-конфиги
|
||||
Арм-механика (editor-model = конфиг → `stageSnap.Model` → снапшот per-арм) **уже работает**. Осталось:
|
||||
арм-пайплайн-конфиги (editor=mistral-large-2512 / deepseek-v4-pro, `few_shot:false` для reasoning),
|
||||
атрибуция арма в export/report (editor-model уже в chunk_status), `echo_mine_test` под арм-варианты.
|
||||
mistral rate_limit — **сдан** (WS1b/WS6 тай).
|
||||
|
||||
### Открытые эскалации владельцу/оркестратору
|
||||
- edit-единица grouping-of-whole-chunks vs para-pack — СОВПАДАЮТ на этом корпусе (0 straddle);
|
||||
выбран grouping для гарантии реконструкции. Требует подтверждения при лендинге.
|
||||
- Реестр §10 (W1.5-UX, DeepSeek ×2 ре-чек 24.07, whole-chapter Select-бюджет) — открыты как в плане.
|
||||
|
|
@ -218,10 +218,16 @@ models:
|
|||
# = $0.5/$1.5 (OpenRouter согласен) → разрешает D19-подозрение ($0.5/$1.5 при $2/$6 = 4×). cached=input
|
||||
# консервативно (cache-read не моделируем; канал B — свежий вход/чанк, hit≈0, потолок не слепнет).
|
||||
price: { input_per_m: 0.5, cached_per_m: 0.5, cache_write_per_m: 0, output_per_m: 1.5 }
|
||||
# WS1 §1б / WS6 gate №4: mistral-large — агрессивный rate-limiter (~48% retry-fails под N-∥ @1.3s;
|
||||
# токен-бакет/конкуренц-кап, тир-зависим — 00-provider-quirks §Транспорт). Пер-модельный семафор
|
||||
# конкуренции волнового раннера ограничивает одновременные вызовы (транспорт-ось, НЕ снапшот).
|
||||
# Значение — консервативный старт; пере-замерить на прод-тире (правило двух направлений). Свап-арм
|
||||
# редактора mistral НЕ идёт через прод-путь без этого guard (ревью-2 F4).
|
||||
rate_limit: { max_concurrency: 2, min_interval_ms: 0 }
|
||||
note: >-
|
||||
Переводчик-дефолт канала B (D19.1): 10/10 чисто на violence, policy 11.06.2026 без запрета adult, не
|
||||
reasoning-модель (эхо-мины нет). Полный wiring канала B — Ф2 (конфиг-слот). Полигон перепроверяет цену (D19.5б).
|
||||
Живьём 2026-07-10: /models ✓, chat 200.
|
||||
reasoning-модель (эхо-мины нет). Свап-арм редактора (WS6, за rate-guard). Полный wiring канала B — Ф2.
|
||||
Полигон перепроверяет цену (D19.5б). Живьём 2026-07-10: /models ✓, chat 200.
|
||||
|
||||
gpt-5-mini: # кандидат-редактор/second-opinion судья (D3) — НЕ в дефолтной цепочке Ф1
|
||||
provider: openai
|
||||
|
|
|
|||
|
|
@ -22,9 +22,20 @@ context:
|
|||
# DeepSeek $0.0028/M — решается расчётом на прототипе).
|
||||
glossary_injection: selective
|
||||
glossary_token_budget: 800
|
||||
stm_depth: 2 # глубина STM в чанках (Фаза 1)
|
||||
overlap_tokens: 200 # перекрытие чанков = read-only контекст (Фаза 1)
|
||||
cache_ttl: "5m"
|
||||
# STMDepth/OverlapTokens сняты (WS2 §2а): carryover/overlap НЕ строится (D39.7/8 — метрики
|
||||
# под полом когезии 0.126); мёртвые кнобы в снапшоте приглашали тихий «re-activate».
|
||||
|
||||
# WS2 (слой 1): бюджет нарезки в ВЫХОДНЫХ (ru) токенах через фертильность (est_out = cjk·CJK +
|
||||
# other·Other); снапшот-folded (segmentationSnap → громкий --resnapshot при правке). Пропуск блока =
|
||||
# ратифицированные zh-ru дефолты (см. LoadPipeline). Крупно-главный арм (edit_ceiling_out>3200)
|
||||
# гейтнут платными Q2a span-судьями (§11) — 3200 = консервативный дефолт.
|
||||
segmentation:
|
||||
draft_budget_out: 1797 # мелкий DRAFT-чанк, ru-выход (→ 56 чанков на 25-гл пере-прогоне)
|
||||
edit_ceiling_out: 3200 # крупная EDIT-единица = глава/группа целых чанков (→ 37 единиц)
|
||||
fertility: # независимо пере-выведено на пере-прогоне: R²=0.9633
|
||||
cjk: 1.1978
|
||||
other: 0.3852
|
||||
|
||||
retries:
|
||||
# Содержательные регенерации после провала гейта ДО эскалации (транспортные
|
||||
|
|
|
|||
|
|
@ -12,9 +12,8 @@ defaults:
|
|||
context:
|
||||
glossary_injection: selective
|
||||
glossary_token_budget: 800
|
||||
stm_depth: 2
|
||||
overlap_tokens: 200
|
||||
cache_ttl: "5m"
|
||||
# STMDepth/OverlapTokens сняты (WS2 §2а — carryover не строим).
|
||||
|
||||
retries:
|
||||
regenerate_before_escalate: 1
|
||||
|
|
|
|||
|
|
@ -130,6 +130,19 @@ type Model struct {
|
|||
// Capabilities overrides this model's provider-default wire shape (D3.1),
|
||||
// field-by-field (model wins). Nil = inherit the provider/kind baseline.
|
||||
Capabilities *CapabilitiesConfig `yaml:"capabilities"`
|
||||
// RateLimit caps how many parallel wave workers may call THIS model at once (WS1 §1б, ревью-2
|
||||
// F4): mistral-large-latest fails ~48% of calls under N-parallelism (a token-bucket / concurrency
|
||||
// cap, tier-dependent — quirks §Транспорт) while grok is 0%, so a model can bound its own
|
||||
// wave concurrency. Optional min_interval paces call STARTS. A TRANSPORT axis — wire-neutral,
|
||||
// NOT snapshot-folded (it never touches the request bytes). Zero = unlimited (the default).
|
||||
RateLimit RateLimit `yaml:"rate_limit"`
|
||||
}
|
||||
|
||||
// RateLimit is the per-model wave-concurrency guard (WS1). max_concurrency 0 = unlimited;
|
||||
// min_interval_ms 0 = no pacing.
|
||||
type RateLimit struct {
|
||||
MaxConcurrency int `yaml:"max_concurrency"`
|
||||
MinIntervalMS int `yaml:"min_interval_ms"`
|
||||
}
|
||||
|
||||
// Price mirrors ledger.ModelPrice in YAML form (USD per 1M tokens).
|
||||
|
|
@ -210,6 +223,9 @@ func LoadModels(path string) (*Models, error) {
|
|||
bad("model %s: non-local models require non-zero input/output prices", name)
|
||||
}
|
||||
validateCapabilities(bad, "model "+name, mod.Capabilities)
|
||||
if mod.RateLimit.MaxConcurrency < 0 || mod.RateLimit.MinIntervalMS < 0 {
|
||||
bad("model %s: rate_limit.max_concurrency/min_interval_ms must be ≥0 (a concurrency cap / pacing interval)", name)
|
||||
}
|
||||
if why := m.echoMineViolation(name); why != "" {
|
||||
bad("model %s on echo-prone provider %s: %s — this re-arms the DeepSeek echo mine "+
|
||||
"(the provider echoes the untranslated CJK source when thinking is OFF; PROGRESS «Ответ Полигону», §2); "+
|
||||
|
|
|
|||
|
|
@ -19,15 +19,43 @@ import (
|
|||
|
||||
// Pipeline is the parsed pipeline-*.yaml.
|
||||
type Pipeline struct {
|
||||
Core string `yaml:"core"` // C0|C1|C2|C3
|
||||
Version int `yaml:"version"`
|
||||
Defaults PipelineDefaults `yaml:"defaults"`
|
||||
Context ContextAssembly `yaml:"context"`
|
||||
Retries Retries `yaml:"retries"`
|
||||
Stages []Stage `yaml:"stages"`
|
||||
Gates Gates `yaml:"gates"`
|
||||
Escal Escalation `yaml:"escalation"`
|
||||
Fanout Fanout `yaml:"fanout"`
|
||||
Core string `yaml:"core"` // C0|C1|C2|C3
|
||||
Version int `yaml:"version"`
|
||||
Defaults PipelineDefaults `yaml:"defaults"`
|
||||
Context ContextAssembly `yaml:"context"`
|
||||
Segmentation Segmentation `yaml:"segmentation"`
|
||||
Retries Retries `yaml:"retries"`
|
||||
Stages []Stage `yaml:"stages"`
|
||||
Gates Gates `yaml:"gates"`
|
||||
Escal Escalation `yaml:"escalation"`
|
||||
Fanout Fanout `yaml:"fanout"`
|
||||
}
|
||||
|
||||
// Segmentation is the WS2 OUTPUT-token chunking budget (слой 1, L2-budget-wrong-unit fix): the
|
||||
// draft-chunk and edit-unit ceilings in ru-OUTPUT tokens, plus the per-pair fertility coefficients
|
||||
// that convert source char-classes to an output-token estimate (est_out = cjk·CJK + other·Other).
|
||||
// Snapshot-folded (segmentationSnap), so a budget/fertility edit is a loud --resnapshot (D30.9).
|
||||
// The char-class CLASSIFIER is the backend's real unicode ranges (EstimateTokens: Han|Hiragana|
|
||||
// Katakana|Hangul) for generality (§0.1) — the coefficients are the only per-pair datum (authored
|
||||
// per project). Zero fields fall to the ratified zh-ru defaults in LoadPipeline.
|
||||
type Segmentation struct {
|
||||
// DraftBudgetOut is the fine DRAFT-chunk ceiling (ru-output tokens, default 1797 → 56 chunks on
|
||||
// the 25-chapter rerun). A draft chunk is the small unit for COGS/coverage/alignment.
|
||||
DraftBudgetOut int `yaml:"draft_budget_out"`
|
||||
// EditCeilingOut is the coarse EDIT-unit ceiling (ru-output tokens, default 3200 → 37 units).
|
||||
// The edit unit is a chapter (or a greedy grouping of whole draft chunks when a chapter exceeds
|
||||
// the ceiling) — the large unit the reflow editor needs for cross-chunk cohesion (D39 п.4). The
|
||||
// large-chapter arm (>3200, up to 8000) is GATED behind paid Q2a span-judges (§11); the ceiling
|
||||
// stays config-tunable but 3200 is the conservative ratified default (span-omission unproven).
|
||||
EditCeilingOut int `yaml:"edit_ceiling_out"`
|
||||
Fertility Fertility `yaml:"fertility"`
|
||||
}
|
||||
|
||||
// Fertility holds the output-token-per-source-char coefficients (WS2). Independently re-derived on
|
||||
// the rerun corpus: cjk=1.1978, other=0.3852 (R²=0.9633). A recompute is a loud --resnapshot.
|
||||
type Fertility struct {
|
||||
CJK float64 `yaml:"cjk"`
|
||||
Other float64 `yaml:"other"`
|
||||
}
|
||||
|
||||
// PipelineDefaults are cross-stage knobs.
|
||||
|
|
@ -39,13 +67,13 @@ type PipelineDefaults struct {
|
|||
}
|
||||
|
||||
// ContextAssembly holds the prompt-layout budgets (Р5-раскладка: стабильный
|
||||
// префикс / волатильный хвост — сборка Фазы 1; поля фиксируются сейчас, чтобы
|
||||
// схема конфига не менялась под ногами «Редакции»).
|
||||
// префикс / волатильный хвост). WS2 (§2а) removed the dead STMDepth/OverlapTokens knobs:
|
||||
// carryover/overlap is NOT built (D39.7/8 — metrics under the 0.126 floor), and leaving
|
||||
// no-op knobs in the wire snapshot invited a silent "re-activate" — so они снесены (a
|
||||
// contextSnap structural change → §8 resnapshot manifest line).
|
||||
type ContextAssembly struct {
|
||||
GlossaryInjection string `yaml:"glossary_injection"` // selective | full_prefix (Р5: обе схемы)
|
||||
GlossaryTokenBudget int `yaml:"glossary_token_budget"`
|
||||
STMDepth int `yaml:"stm_depth"`
|
||||
OverlapTokens int `yaml:"overlap_tokens"`
|
||||
CacheTTL string `yaml:"cache_ttl"` // per-stage TTL override — Фаза 1
|
||||
}
|
||||
|
||||
|
|
@ -285,6 +313,27 @@ func LoadPipeline(path string, models *Models) (*Pipeline, error) {
|
|||
if p.Fanout.Candidates <= 0 {
|
||||
p.Fanout.Candidates = 1
|
||||
}
|
||||
// WS2 segmentation defaults (ratified zh-ru, §2в): a config that omits the block gets the
|
||||
// output-token budget + independently-re-derived fertility. Zero/negative → default.
|
||||
if p.Segmentation.DraftBudgetOut <= 0 {
|
||||
p.Segmentation.DraftBudgetOut = 1797
|
||||
}
|
||||
if p.Segmentation.EditCeilingOut <= 0 {
|
||||
p.Segmentation.EditCeilingOut = 3200
|
||||
}
|
||||
if p.Segmentation.Fertility.CJK <= 0 {
|
||||
p.Segmentation.Fertility.CJK = 1.1978
|
||||
}
|
||||
if p.Segmentation.Fertility.Other <= 0 {
|
||||
p.Segmentation.Fertility.Other = 0.3852
|
||||
}
|
||||
// An edit unit is a grouping of WHOLE draft chunks, so the edit ceiling must be ≥ the draft
|
||||
// budget — otherwise a single draft chunk already exceeds the unit ceiling and every unit is
|
||||
// one chunk (the decoupling collapses). Loud config error, not a silent degenerate segmentation.
|
||||
if p.Segmentation.EditCeilingOut < p.Segmentation.DraftBudgetOut {
|
||||
bad("segmentation.edit_ceiling_out (%d) must be ≥ draft_budget_out (%d) — an edit unit groups whole draft chunks (WS2)",
|
||||
p.Segmentation.EditCeilingOut, p.Segmentation.DraftBudgetOut)
|
||||
}
|
||||
switch p.Context.GlossaryInjection {
|
||||
case "", "selective", "full_prefix":
|
||||
default:
|
||||
|
|
|
|||
135
backend/internal/pipeline/banknote.go
Normal file
135
backend/internal/pipeline/banknote.go
Normal file
|
|
@ -0,0 +1,135 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// banknote.go: the banknote-v1 in-band footnote channel (WS4, план §4 / research/20 §B3 / exp16
|
||||
// banknote.py — RATIFIED D39.10 as the dst-delivery channel). The translator, AFTER the translation,
|
||||
// MAY emit a versioned separator + tab-delimited term lines for NEW terms only (not in the injected
|
||||
// glossary). This file is the pure, deterministic PARSER + SLICER — the wire-invariant core; the
|
||||
// integration seams (slice BEFORE classifyOutput, derived checkpoint, snapshot fold, telemetry —
|
||||
// план §4(а) 12 points) consume these functions. A faithful Go↔Python port (banknote.py is the
|
||||
// reference): same separator, same tolerant field split, same Han-in-src rule, same truncation
|
||||
// tolerance. Pin (ws4_banknote_verify.py): re-parse of 95 saved lines = 0 parse_fail / 0 truncated.
|
||||
|
||||
// bankSeparator marks the start of the banknote block. Chosen to (a) not occur in natural prose and
|
||||
// (b) NOT be a "Примечание/Сноска/Комментарий" trailing-note word the sanitizer reserves (§B3-1), so
|
||||
// the two channels never collide. Changing it is a wire/parser change → a loud --resnapshot (§4в).
|
||||
const bankSeparator = "⟦TM-BANK-v1⟧"
|
||||
|
||||
// bankMaxLines is the per-chunk banknote line budget (§B3-5, feeds the max_tokens sizing at
|
||||
// integration point 5). The parser does not enforce it (a model over-emitting is tolerated and
|
||||
// merely over-counted); it bounds the prompt-side budget.
|
||||
const bankMaxLines = 12
|
||||
|
||||
// bankParserVersion versions the split/parse algorithm — the verdict-axis component folded into
|
||||
// banknoteSnap{enabled, parser_version} (§4в/§4а point 6), mirroring sanitizerSnap: a parser change
|
||||
// re-resolves the stripped draft, so it must be a loud --resnapshot even without a prompt edit.
|
||||
const bankParserVersion = "banknote-v1"
|
||||
|
||||
// bankTypeOK is the accepted type set; anything else falls back to "term" (a benign default).
|
||||
var bankTypeOK = map[string]bool{"name": true, "place": true, "title": true, "term": true, "nickname": true}
|
||||
|
||||
// bankFieldSplit tolerates a tab, a run of ≥2 spaces, or a pipe (optionally whitespace-padded) as the
|
||||
// field delimiter — the exact tolerance of banknote.py so a model that emits spaces instead of a real
|
||||
// TAB still parses. (Go \s is ASCII-whitespace; banknote lines are tab/space-delimited, so the parity
|
||||
// with Python's unicode \s around the pipe is exact on the term corpus.)
|
||||
var bankFieldSplit = regexp.MustCompile(`\t| {2,}|\s*\|\s*`)
|
||||
|
||||
// bankEntry is one parsed candidate line (evidence for W1.5 §C; NOT written to the store until owner
|
||||
// signoff). Dst is the model's proposed translation — the direct dst delivery the co-occurrence
|
||||
// miner could not extract (蛊→гу, non-seed 龙公→Лун Гун).
|
||||
type bankEntry struct {
|
||||
Src string
|
||||
Dst string
|
||||
Type string
|
||||
}
|
||||
|
||||
// bankFlags is the per-chunk telemetry surfaced loud at integration point 10.
|
||||
type bankFlags struct {
|
||||
NLines int // accepted (well-formed) banknote lines
|
||||
ParseFail bool // any malformed residual line (non-Han src / <2 fields) that was NOT a tolerated truncation
|
||||
Truncated bool // the LAST line was cut by generation length (tolerated, not a parse fail)
|
||||
}
|
||||
|
||||
// splitBanknote slices the banknote block off the raw model output BEFORE any gate/editor (integration
|
||||
// point 1). Returns (clean_translation, raw_block). No separator → the whole (right-trimmed) output is
|
||||
// the clean translation and the block is "". Faithful to banknote.py: clean = output[:idx].rstrip(),
|
||||
// block = output[idx+len(SEP):].strip("\n").
|
||||
func splitBanknote(output string) (clean, block string) {
|
||||
idx := strings.Index(output, bankSeparator)
|
||||
if idx < 0 {
|
||||
return strings.TrimRightFunc(output, unicode.IsSpace), ""
|
||||
}
|
||||
clean = strings.TrimRightFunc(output[:idx], unicode.IsSpace)
|
||||
block = strings.Trim(output[idx+len(bankSeparator):], "\n")
|
||||
return clean, block
|
||||
}
|
||||
|
||||
// hasBankSrcHan reports whether src contains a CJK ideograph in the exact [㐀-鿿] range (U+3400–U+9FFF)
|
||||
// banknote.py checks — a zh→ru channel line whose src has no Han is malformed. Kept as the exact
|
||||
// Python range (NOT unicode.Han, which is wider) for byte-faithful parity.
|
||||
func hasBankSrcHan(src string) bool {
|
||||
for _, r := range src {
|
||||
if r >= 0x3400 && r <= 0x9FFF {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// parseBanknote parses the tab-delimited block (integration seam feeding §C evidence + telemetry).
|
||||
// Tolerant of a TRUNCATED final line when truncatedGeneration is set (§B3-5): a short last line under
|
||||
// truncation is flagged banknote_truncated, not counted as a parse fail. Any other malformed line
|
||||
// (fewer than 2 fields, or a src with no Han) sets banknote_parse_fail. Deterministic, no time/rand.
|
||||
func parseBanknote(block string, truncatedGeneration bool) ([]bankEntry, bankFlags) {
|
||||
var entries []bankEntry
|
||||
var flags bankFlags
|
||||
if strings.TrimSpace(block) == "" {
|
||||
return entries, flags
|
||||
}
|
||||
var lines []string
|
||||
for _, ln := range strings.Split(block, "\n") {
|
||||
if strings.TrimSpace(ln) != "" {
|
||||
lines = append(lines, ln)
|
||||
}
|
||||
}
|
||||
bad := 0
|
||||
for i, ln := range lines {
|
||||
raw := bankFieldSplit.Split(strings.TrimSpace(ln), -1)
|
||||
var parts []string
|
||||
for _, p := range raw {
|
||||
if p = strings.TrimSpace(p); p != "" {
|
||||
parts = append(parts, p)
|
||||
}
|
||||
}
|
||||
if len(parts) < 2 {
|
||||
// A short LAST line under a truncated generation is a tolerated cut, not a failure.
|
||||
if i == len(lines)-1 && truncatedGeneration {
|
||||
flags.Truncated = true
|
||||
continue
|
||||
}
|
||||
bad++
|
||||
continue
|
||||
}
|
||||
src, dst := parts[0], parts[1]
|
||||
typ := "term"
|
||||
if len(parts) >= 3 {
|
||||
typ = strings.ToLower(parts[2])
|
||||
}
|
||||
if !bankTypeOK[typ] {
|
||||
typ = "term"
|
||||
}
|
||||
if !hasBankSrcHan(src) {
|
||||
bad++
|
||||
continue
|
||||
}
|
||||
entries = append(entries, bankEntry{Src: src, Dst: dst, Type: typ})
|
||||
}
|
||||
flags.NLines = len(entries)
|
||||
flags.ParseFail = bad > 0
|
||||
return entries, flags
|
||||
}
|
||||
101
backend/internal/pipeline/banknote_test.go
Normal file
101
backend/internal/pipeline/banknote_test.go
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// banknote_test.go pins the WS4 §4(д) parser invariants (byte-faithful port of exp16 banknote.py).
|
||||
// The 95-line corpus pin (0 parse_fail / 0 truncated over 14 saved raw outputs) is the domain of the
|
||||
// $0 reference eval/design11/ws4_banknote_verify.py; these unit tests cover the parser CONTRACT.
|
||||
|
||||
func TestSplitBanknoteNoSeparatorIsAllClean(t *testing.T) {
|
||||
out := "Судзуки шёл по коридору. \n"
|
||||
clean, block := splitBanknote(out)
|
||||
if clean != "Судзуки шёл по коридору." || block != "" {
|
||||
t.Fatalf("no-SEP: clean=%q block=%q, want the whole right-trimmed output and empty block", clean, block)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitBanknoteSlicesBlock(t *testing.T) {
|
||||
out := "Перевод... последнее предложение.\n\n" + bankSeparator + "\n方源\tФан Юань\tname\n蛊师\tгу-мастер\ttitle"
|
||||
clean, block := splitBanknote(out)
|
||||
if !strings.HasSuffix(clean, "последнее предложение.") {
|
||||
t.Fatalf("clean tail wrong: %q", clean)
|
||||
}
|
||||
if strings.Contains(clean, bankSeparator) || strings.Contains(clean, "方源") {
|
||||
t.Fatalf("the banknote block leaked into the clean translation: %q", clean)
|
||||
}
|
||||
ents, flags := parseBanknote(block, false)
|
||||
if len(ents) != 2 || flags.ParseFail || flags.Truncated || flags.NLines != 2 {
|
||||
t.Fatalf("parse = %+v flags=%+v, want 2 clean entries", ents, flags)
|
||||
}
|
||||
if ents[0].Src != "方源" || ents[0].Dst != "Фан Юань" || ents[0].Type != "name" {
|
||||
t.Fatalf("entry[0] = %+v", ents[0])
|
||||
}
|
||||
if ents[1].Type != "title" {
|
||||
t.Fatalf("entry[1] type = %q, want title", ents[1].Type)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBanknoteTolerantFieldSplit(t *testing.T) {
|
||||
// tab, ≥2 spaces, and pipe are all accepted delimiters (a model that emits spaces instead of a
|
||||
// real TAB still parses). type falls back to "term" when absent or unknown.
|
||||
block := "方源\tФан Юань\tname\n蛊师 гу-мастер title\n青茅山 | гора Цинмао | place\n古月\tГу Юэ"
|
||||
ents, flags := parseBanknote(block, false)
|
||||
if flags.ParseFail {
|
||||
t.Fatalf("tolerant split should not fail: %+v", flags)
|
||||
}
|
||||
if len(ents) != 4 {
|
||||
t.Fatalf("want 4 entries across tab/space/pipe delimiters, got %d: %+v", len(ents), ents)
|
||||
}
|
||||
if ents[3].Type != "term" { // no type field → default term
|
||||
t.Fatalf("missing type must default to term, got %q", ents[3].Type)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBanknoteNonHanSrcIsBad(t *testing.T) {
|
||||
// A zh→ru channel line whose src has no Han ideograph is malformed → parse_fail.
|
||||
block := "方源\tФан Юань\tname\nRoseanne\tРозанна\tname"
|
||||
ents, flags := parseBanknote(block, false)
|
||||
if !flags.ParseFail {
|
||||
t.Fatalf("a non-Han src must set parse_fail")
|
||||
}
|
||||
if len(ents) != 1 || ents[0].Src != "方源" {
|
||||
t.Fatalf("only the Han-src line should survive, got %+v", ents)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBanknoteTruncationTolerated(t *testing.T) {
|
||||
// The LAST line cut by generation length is tolerated (banknote_truncated), not a parse fail —
|
||||
// but ONLY under truncated_generation; the same short line otherwise IS a parse fail.
|
||||
block := "方源\tФан Юань\tname\n蛊" // last line has no dst
|
||||
entsT, flagsT := parseBanknote(block, true)
|
||||
if !flagsT.Truncated || flagsT.ParseFail || len(entsT) != 1 {
|
||||
t.Fatalf("truncated=true: want 1 entry + truncated flag + no parse_fail, got %+v %+v", entsT, flagsT)
|
||||
}
|
||||
entsF, flagsF := parseBanknote(block, false)
|
||||
if flagsF.Truncated || !flagsF.ParseFail || len(entsF) != 1 {
|
||||
t.Fatalf("truncated=false: the short last line must be a parse_fail, got %+v %+v", entsF, flagsF)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseBanknoteEmptyBlock(t *testing.T) {
|
||||
ents, flags := parseBanknote("", false)
|
||||
if len(ents) != 0 || flags.ParseFail || flags.Truncated || flags.NLines != 0 {
|
||||
t.Fatalf("empty block must yield no entries and clean flags, got %+v %+v", ents, flags)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBankSeparatorIsNotANoteWord(t *testing.T) {
|
||||
// The separator must not be a trailing-note word the sanitizer reserves (§B3-1) — the two
|
||||
// channels must never collide.
|
||||
for _, note := range []string{"Примечание", "Сноска", "Комментарий", "Note", "TN"} {
|
||||
if strings.Contains(bankSeparator, note) {
|
||||
t.Fatalf("separator %q collides with the reserved note-word %q", bankSeparator, note)
|
||||
}
|
||||
}
|
||||
if !strings.HasPrefix(bankSeparator, "⟦") {
|
||||
t.Fatalf("separator lost its unusual delimiter: %q", bankSeparator)
|
||||
}
|
||||
}
|
||||
|
|
@ -97,6 +97,15 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
|
|||
return nil, err
|
||||
}
|
||||
|
||||
// W0: eager-build every reachable client BEFORE any wave goroutine, so the clients map is
|
||||
// read-only in the waves (r.client is lock-free, a miss is loud — closes the D12 lazy-init race).
|
||||
if err := r.buildClients(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// W0: build the per-model rate-guards (transport axis, read-only in the waves; a no-op until a
|
||||
// model configures rate_limit — WS1 §1б, the mistral-arm precondition WS6).
|
||||
r.buildRateGuards()
|
||||
|
||||
// Persist captured ruby readings (idempotent; consumed by seedGlossary below into
|
||||
// auto glossary candidates — шаг 4). Never injected into a prompt here (§7d); a
|
||||
// resume re-persists the same rows at $0.
|
||||
|
|
@ -122,7 +131,7 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
|
|||
return nil, fmt.Errorf("pipeline: upsert snapshot %.12s: %w", snapID, err)
|
||||
}
|
||||
|
||||
chunks := SplitChunks(doc.Chapters)
|
||||
chunks := SplitChunks(doc.Chapters, r.segBudget())
|
||||
if len(chunks) == 0 {
|
||||
return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,19 +7,21 @@ import (
|
|||
|
||||
// chunker.go: the шаг-3a SOURCE segmenter. It turns the per-chapter normalized
|
||||
// text produced by ingest.go into an ordered list of (chapter, chunk) units for
|
||||
// the runner loop. It replaced the Веха-2 placeholder (chapter split on \f +
|
||||
// blank-line paragraph packing to a CHAR budget) with real segmentation: pack
|
||||
// whole paragraphs to a ~1–2k-TOKEN budget, and when a single paragraph exceeds
|
||||
// the budget descend to SENTENCE boundaries — never splitting a sentence.
|
||||
// the runner loop. WS2 (слой 1) moved the packing budget to OUTPUT (ru) tokens via
|
||||
// the fertility estimate (est_out = f_cjk·cjk + f_other·other over source char-classes),
|
||||
// decoupling draft sizing (small chunk, COGS/coverage/alignment) from the EDIT unit
|
||||
// (coarse chapter-scale unit for cross-chunk cohesion): pack whole paragraphs to
|
||||
// DraftBudgetOut, and when a single paragraph exceeds it descend to SENTENCE boundaries
|
||||
// (never splitting a sentence; a lone over-budget sentence is a passthrough
|
||||
// OversizedSentence chunk). Edit units then group WHOLE draft chunks to EditCeilingOut.
|
||||
//
|
||||
// Contract kept intact for the loop (bookrun.go/chunkrun.go depend on it): SplitChunks emits
|
||||
// Chunk{Chapter,ChunkIdx,Text}; ChunkIdx resets to 0 per chapter; the function is
|
||||
// a PURE, deterministic function of its input (no time/rand, no map iteration).
|
||||
// Its behavior is versioned by chunkerVersion (render.go), folded into the job
|
||||
// snapshot — so changing the segmentation is a loud --resnapshot re-translation,
|
||||
// never a silent cache miss (§3.4/R6/D5.2). The chunk target is a CONST covered by
|
||||
// that version (Р2 = code); were it ever made config-tunable it MUST be folded into
|
||||
// the top-level snapshot like coverageSnap (§7d), not left to the version string.
|
||||
// Chunk{Chapter,ChunkIdx,Text,…}; ChunkIdx resets to 0 per chapter; the new fields (EstOut,
|
||||
// OversizedSentence, EditUnitID) are ADDITIVE; the function is a PURE, deterministic function of
|
||||
// (chapters, SegBudget) (no time/rand, no map iteration). Its behavior is versioned by
|
||||
// chunkerVersion (render.go); the SegBudget is snapshot-folded via segmentationSnap — so changing
|
||||
// the segmentation OR the budget/fertility is a loud --resnapshot re-translation, never a silent
|
||||
// cache miss (§3.4/R6/D5.2/D30.9).
|
||||
//
|
||||
// IMPORTANT: this sentence splitter is a SEPARATE, independent piece of logic from
|
||||
// coverage.go's splitSentences. That one is a bit-for-bit port of the Python oracle
|
||||
|
|
@ -33,84 +35,113 @@ type Chunk struct {
|
|||
Chapter int // 1-based
|
||||
ChunkIdx int // 0-based within its chapter
|
||||
Text string
|
||||
// EstOut is the fertility-estimated OUTPUT (ru) token cost of this chunk (WS2, слой 1):
|
||||
// est_out = fert.CJK·cjk_src + fert.Other·other_src over the source char-classes. It is the
|
||||
// unit the draft budget packs against (DraftBudgetOut), decoupling draft sizing from the
|
||||
// old input-token heuristic (L2-budget-wrong-unit). Additive field; the {Chapter,ChunkIdx,Text}
|
||||
// contract the loop/ingest/status depend on is untouched.
|
||||
EstOut float64
|
||||
// OversizedSentence marks a chunk that is a SINGLE source sentence whose est_out alone exceeds
|
||||
// DraftBudgetOut (WS2 §2б). It is passthrough (never clause-split — clause splitting is
|
||||
// boundary machinery, anti-scope), and the flag makes the un-budgetable unit OBSERVABLE
|
||||
// ("деградация не молчит") rather than silently oversized.
|
||||
OversizedSentence bool
|
||||
// EditUnitID is the 0-based book-global id of the coarse EDIT unit this draft chunk belongs to
|
||||
// (WS2 decoupling: draft = small chunk, edit = large unit). An edit unit is a greedy grouping of
|
||||
// WHOLE draft chunks up to EditCeilingOut (per chapter), so W2's editor reads a unit's draft as
|
||||
// the concatenation of its member chunks' W1 outputs — clean reconstruction by construction (no
|
||||
// straddle; verified: grouping whole chunks reproduces the paragraph-packed count 37 with 0
|
||||
// straddle on the rerun corpus). Draft chunks never cross an edit-unit boundary.
|
||||
EditUnitID int
|
||||
}
|
||||
|
||||
// targetChunkTokens is the packing budget: paragraphs (or, inside an oversize
|
||||
// paragraph, sentences) are greedily packed until the WHOLE chunk's estimated
|
||||
// tokens (EstimateTokens' formula, applied once — see appendChapterChunks) would
|
||||
// exceed this. ~1500 sits in the plan's "1–2k токенов по границам абзацев" corridor
|
||||
// (02-mvp Фаза 1); a chunk of ordinary prose lands well above the coverage gate's
|
||||
// 500-char floor, so the excision gate is not silently disabled (§7b). A chunk can
|
||||
// still fall under the floor only when a whole chapter (or its tail) is genuinely
|
||||
// that short — a documented boundary. A const, not config: the size is a code
|
||||
// segmentation rule (Р2), covered by chunkerVersion.
|
||||
const targetChunkTokens = 1500
|
||||
// SegBudget is the resolved output-token segmentation budget (WS2 §2в): the draft-chunk and
|
||||
// edit-unit ceilings in ru-OUTPUT tokens, plus the per-pair fertility coefficients that convert
|
||||
// source char-classes to an output-token estimate. Resolved from config.Segmentation by the runner
|
||||
// and snapshot-folded via segmentationSnap, so a budget/fertility edit is a loud --resnapshot
|
||||
// (D30.9). The classifier (tokenClassCounts) uses the REAL unicode.RangeTable of EstimateTokens
|
||||
// (Han|Hiragana|Katakana|Hangul) for generality (§0.1) — NOT script-specific ord ranges.
|
||||
type SegBudget struct {
|
||||
DraftBudgetOut float64 // draft-chunk ceiling, ru-output tokens (default 1797)
|
||||
EditCeilingOut float64 // edit-unit ceiling, ru-output tokens (default 3200, gated arm at 8000)
|
||||
FertCJK float64 // output tokens per CJK source char (default 1.1978)
|
||||
FertOther float64 // output tokens per non-CJK/non-space source char (default 0.3852)
|
||||
}
|
||||
|
||||
// estOut applies the fertility formula to pre-counted source char-classes — identical to
|
||||
// est_out over the joined chunk text (whitespace is class-free, as tokenClassCounts drops it).
|
||||
func (s SegBudget) estOut(cjk, other int) float64 {
|
||||
return s.FertCJK*float64(cjk) + s.FertOther*float64(other)
|
||||
}
|
||||
|
||||
// SplitChunks segments the ordered, per-chapter NORMALIZED text of a Document
|
||||
// (ingest.go already applied NormalizeSource and split chapters) into an ordered
|
||||
// chunk list. Chapters that yield no text (blank/whitespace-only, e.g. an epub
|
||||
// cover/nav page) do NOT consume a chapter number, so numbering stays dense over
|
||||
// real content (Фаза-0 backward compat). Fully deterministic.
|
||||
func SplitChunks(chapters []string) []Chunk {
|
||||
// real content (Фаза-0 backward compat). Fully deterministic. Each chunk carries its
|
||||
// EstOut + OversizedSentence flag + EditUnitID (WS2); the edit-unit id is monotone across
|
||||
// the book so a W2 unit is uniquely addressable.
|
||||
func SplitChunks(chapters []string, seg SegBudget) []Chunk {
|
||||
var out []Chunk
|
||||
chapterNo := 0
|
||||
editUnitID := 0
|
||||
for _, chapText := range chapters {
|
||||
paras := splitParagraphs(chapText)
|
||||
if len(paras) == 0 {
|
||||
continue // an empty chapter does not consume a chapter number
|
||||
}
|
||||
chapterNo++
|
||||
out = appendChapterChunks(out, chapterNo, paras)
|
||||
chapterChunks := chapterDraftChunks(chapterNo, paras, seg)
|
||||
assignEditUnits(chapterChunks, seg, &editUnitID)
|
||||
out = append(out, chapterChunks...)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// appendChapterChunks packs one chapter's paragraphs into chunks and appends them.
|
||||
// Rule: pack whole paragraphs while the running token estimate stays within
|
||||
// targetChunkTokens (prefer paragraph boundaries); a paragraph that alone exceeds
|
||||
// the budget is flushed on its own and then SENTENCE-packed (never splitting a
|
||||
// sentence; a single sentence over budget is its own chunk).
|
||||
//
|
||||
// The running budget is tracked as ADDITIVE char-class counts (CJK vs other),
|
||||
// applying EstimateTokens' formula (and its 16-token floor) ONCE to the whole
|
||||
// chunk — NOT by summing per-unit EstimateTokens. Summing per-unit floors every
|
||||
// tiny paragraph up to 16 tokens, so a dialogue chapter of many short lines would
|
||||
// flush a chunk at ~⅓ its true token content, needlessly dropping real content
|
||||
// under the coverage gate's 500-char floor (self-review finding). Counting classes
|
||||
// and flooring once reproduces EstimateTokens(joined text) exactly (the "\n\n"
|
||||
// separators are whitespace, which EstimateTokens ignores).
|
||||
func appendChapterChunks(out []Chunk, chapterNo int, paras []string) []Chunk {
|
||||
// chapterDraftChunks packs one chapter's paragraphs into DRAFT chunks (the fine tiling). Rule:
|
||||
// pack whole paragraphs while the running OUTPUT-token estimate stays within DraftBudgetOut
|
||||
// (prefer paragraph boundaries); a paragraph that alone exceeds the budget is flushed on its own
|
||||
// and then SENTENCE-packed (never splitting a sentence; a single sentence over budget is its own
|
||||
// OversizedSentence chunk). The running budget is tracked as ADDITIVE char-class counts (CJK vs
|
||||
// other), applying est_out ONCE to the whole chunk — the "\n\n" separators are whitespace, which
|
||||
// tokenClassCounts drops, so counting classes and estimating once reproduces est_out(joined text)
|
||||
// exactly. Each emitted chunk gets its ChunkIdx (per-chapter) + EstOut; EditUnitID is set later.
|
||||
func chapterDraftChunks(chapterNo int, paras []string, seg SegBudget) []Chunk {
|
||||
var out []Chunk
|
||||
chunkIdx := 0
|
||||
emit := func(text string, oversized bool, cjk, other int) {
|
||||
if t := strings.TrimSpace(text); t != "" {
|
||||
out = append(out, Chunk{
|
||||
Chapter: chapterNo, ChunkIdx: chunkIdx, Text: t,
|
||||
EstOut: seg.estOut(cjk, other), OversizedSentence: oversized,
|
||||
})
|
||||
chunkIdx++
|
||||
}
|
||||
}
|
||||
var buf []string // paragraphs accumulated for the current chunk
|
||||
var bufCJK, bufOther int
|
||||
flush := func() {
|
||||
if len(buf) == 0 {
|
||||
return
|
||||
}
|
||||
if text := strings.TrimSpace(strings.Join(buf, "\n\n")); text != "" {
|
||||
out = append(out, Chunk{Chapter: chapterNo, ChunkIdx: chunkIdx, Text: text})
|
||||
chunkIdx++
|
||||
}
|
||||
emit(strings.Join(buf, "\n\n"), false, bufCJK, bufOther)
|
||||
buf = buf[:0]
|
||||
bufCJK, bufOther = 0, 0
|
||||
}
|
||||
for _, p := range paras {
|
||||
pCJK, pOther := tokenClassCounts(p)
|
||||
if estTokensFrom(pCJK, pOther) > targetChunkTokens {
|
||||
if seg.estOut(pCJK, pOther) > seg.DraftBudgetOut {
|
||||
// Oversize paragraph: close any open chunk at this paragraph boundary,
|
||||
// then split the paragraph at sentence boundaries into its own chunks.
|
||||
flush()
|
||||
for _, sub := range packSentences(p) {
|
||||
if text := strings.TrimSpace(sub); text != "" {
|
||||
out = append(out, Chunk{Chapter: chapterNo, ChunkIdx: chunkIdx, Text: text})
|
||||
chunkIdx++
|
||||
}
|
||||
for _, sub := range packSentences(p, seg) {
|
||||
emit(sub.text, sub.oversizedSentence, sub.cjk, sub.other)
|
||||
}
|
||||
continue
|
||||
}
|
||||
// Normal paragraph: start a new chunk if adding it would exceed the budget
|
||||
// and the current chunk is non-empty (never split a paragraph here).
|
||||
if len(buf) > 0 && estTokensFrom(bufCJK+pCJK, bufOther+pOther) > targetChunkTokens {
|
||||
if len(buf) > 0 && seg.estOut(bufCJK+pCJK, bufOther+pOther) > seg.DraftBudgetOut {
|
||||
flush()
|
||||
}
|
||||
buf = append(buf, p)
|
||||
|
|
@ -120,28 +151,67 @@ func appendChapterChunks(out []Chunk, chapterNo int, paras []string) []Chunk {
|
|||
return out
|
||||
}
|
||||
|
||||
// packSentences splits one oversize paragraph into sentence segments (lossless
|
||||
// tiling — concatenation reproduces the paragraph) and packs consecutive segments
|
||||
// into ≤ targetChunkTokens sub-chunks, never splitting a sentence. A single
|
||||
// sentence larger than the budget becomes its own (oversize) sub-chunk — the only
|
||||
// way to honor "never split a sentence". Returns the sub-chunk texts in order. Uses
|
||||
// the same additive char-class budget as appendChapterChunks (floor applied once).
|
||||
func packSentences(paragraph string) []string {
|
||||
// assignEditUnits groups a chapter's already-built DRAFT chunks into EDIT units — greedy packing of
|
||||
// WHOLE draft chunks until the accumulated est_out would exceed EditCeilingOut — and stamps each
|
||||
// chunk's EditUnitID (monotone across the book via *nextID). Because units are formed from whole
|
||||
// draft chunks, a unit's source/draft span is exactly the concatenation of its member chunks (no
|
||||
// straddle) and W2 reconstructs the unit's draft losslessly. A single draft chunk larger than the
|
||||
// ceiling (an OversizedSentence, or a chapter whose first chunk already exceeds the ceiling) is its
|
||||
// own unit — grouping never splits a chunk (never-split-below-chunk mirrors never-split-paragraph).
|
||||
func assignEditUnits(chunks []Chunk, seg SegBudget, nextID *int) {
|
||||
acc := 0.0
|
||||
started := false
|
||||
for i := range chunks {
|
||||
if started && acc+chunks[i].EstOut > seg.EditCeilingOut {
|
||||
*nextID++ // close the current unit at this chunk boundary
|
||||
acc = 0
|
||||
}
|
||||
chunks[i].EditUnitID = *nextID
|
||||
acc += chunks[i].EstOut
|
||||
started = true
|
||||
}
|
||||
if started {
|
||||
*nextID++ // advance past the last unit so the next chapter starts a fresh id
|
||||
}
|
||||
}
|
||||
|
||||
// subChunk is one sentence-packed sub-chunk with its char-class counts and the oversized-sentence
|
||||
// flag (a single sentence whose est_out alone exceeds DraftBudgetOut).
|
||||
type subChunk struct {
|
||||
text string
|
||||
cjk, other int
|
||||
oversizedSentence bool
|
||||
}
|
||||
|
||||
// packSentences splits one oversize paragraph into sentence segments (lossless tiling —
|
||||
// concatenation reproduces the paragraph) and packs consecutive segments into ≤ DraftBudgetOut
|
||||
// (est_out) sub-chunks, never splitting a sentence. A single sentence larger than the budget
|
||||
// becomes its own sub-chunk flagged OversizedSentence (passthrough, never clause-split — the plan's
|
||||
// §2б decision: "деградация не молчит" is satisfied by the FLAG, not emergency clause machinery).
|
||||
func packSentences(paragraph string, seg SegBudget) []subChunk {
|
||||
segs := splitSourceSentences(paragraph)
|
||||
var chunks []string
|
||||
var chunks []subChunk
|
||||
var buf strings.Builder
|
||||
var bufCJK, bufOther int
|
||||
flush := func() {
|
||||
if buf.Len() == 0 {
|
||||
return
|
||||
}
|
||||
chunks = append(chunks, buf.String())
|
||||
chunks = append(chunks, subChunk{text: buf.String(), cjk: bufCJK, other: bufOther})
|
||||
buf.Reset()
|
||||
bufCJK, bufOther = 0, 0
|
||||
}
|
||||
for _, s := range segs {
|
||||
sCJK, sOther := tokenClassCounts(s)
|
||||
if buf.Len() > 0 && estTokensFrom(bufCJK+sCJK, bufOther+sOther) > targetChunkTokens {
|
||||
// A lone sentence over budget: flush what precedes it, then emit it as its own
|
||||
// OversizedSentence sub-chunk (never split a sentence, and never merge it with a
|
||||
// neighbour so the oversized flag stays attributable to the single sentence).
|
||||
if seg.estOut(sCJK, sOther) > seg.DraftBudgetOut {
|
||||
flush()
|
||||
chunks = append(chunks, subChunk{text: s, cjk: sCJK, other: sOther, oversizedSentence: true})
|
||||
continue
|
||||
}
|
||||
if buf.Len() > 0 && seg.estOut(bufCJK+sCJK, bufOther+sOther) > seg.DraftBudgetOut {
|
||||
flush()
|
||||
}
|
||||
buf.WriteString(s)
|
||||
|
|
@ -170,16 +240,6 @@ func tokenClassCounts(s string) (cjk, other int) {
|
|||
return cjk, other
|
||||
}
|
||||
|
||||
// estTokensFrom applies EstimateTokens' formula (cjk + other/3, floored at 16) to
|
||||
// pre-counted class totals — identical to EstimateTokens over the joined text.
|
||||
func estTokensFrom(cjk, other int) int {
|
||||
est := cjk + other/3
|
||||
if est < 16 {
|
||||
est = 16
|
||||
}
|
||||
return est
|
||||
}
|
||||
|
||||
// splitParagraphs breaks a chapter into paragraphs on blank lines and trims each,
|
||||
// dropping empties. Deterministic; whitespace-only paragraphs vanish. (Ingest emits
|
||||
// a blank line between block-level epub elements, so paragraphs survive extraction.)
|
||||
|
|
|
|||
|
|
@ -8,43 +8,72 @@ import (
|
|||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// testSeg is the ratified WS2 output-token budget (zh-ru defaults) used by the chunker tests.
|
||||
func testSeg() SegBudget {
|
||||
return SegBudget{DraftBudgetOut: 1797, EditCeilingOut: 3200, FertCJK: 1.1978, FertOther: 0.3852}
|
||||
}
|
||||
|
||||
// stripMeta zeroes the WS2 additive fields (EstOut/OversizedSentence/EditUnitID) so a
|
||||
// {Chapter,ChunkIdx,Text}-level DeepEqual still expresses the intent of the pre-WS2 assertions.
|
||||
func stripMeta(cs []Chunk) []Chunk {
|
||||
out := make([]Chunk, len(cs))
|
||||
for i, c := range cs {
|
||||
out[i] = Chunk{Chapter: c.Chapter, ChunkIdx: c.ChunkIdx, Text: c.Text}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestSplitChunksSingleParagraph(t *testing.T) {
|
||||
got := SplitChunks([]string{"静かな図書館の朝。"})
|
||||
got := SplitChunks([]string{"静かな図書館の朝。"}, testSeg())
|
||||
want := []Chunk{{Chapter: 1, ChunkIdx: 0, Text: "静かな図書館の朝。"}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||
t.Fatalf("single paragraph = %+v, want %+v", got, want)
|
||||
}
|
||||
// WS2: the additive fields are populated — est_out>0 and a single unit id.
|
||||
if got[0].EstOut <= 0 || got[0].EditUnitID != 0 || got[0].OversizedSentence {
|
||||
t.Fatalf("WS2 fields = est_out %.3f unit %d oversized %v", got[0].EstOut, got[0].EditUnitID, got[0].OversizedSentence)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitChunksChapters(t *testing.T) {
|
||||
got := SplitChunks([]string{"Глава один.", "Глава два.", "Глава три."})
|
||||
got := SplitChunks([]string{"Глава один.", "Глава два.", "Глава три."}, testSeg())
|
||||
want := []Chunk{
|
||||
{Chapter: 1, ChunkIdx: 0, Text: "Глава один."},
|
||||
{Chapter: 2, ChunkIdx: 0, Text: "Глава два."},
|
||||
{Chapter: 3, ChunkIdx: 0, Text: "Глава три."},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||
t.Fatalf("chapters = %+v, want %+v", got, want)
|
||||
}
|
||||
// WS2: each chapter is its own edit unit → distinct monotone EditUnitIDs 0,1,2.
|
||||
for i, c := range got {
|
||||
if c.EditUnitID != i {
|
||||
t.Fatalf("chapter %d chunk edit_unit_id = %d, want %d (each short chapter is its own unit)", c.Chapter, c.EditUnitID, i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitChunksPacksToTarget(t *testing.T) {
|
||||
// Two paragraphs that together exceed the TOKEN target (each ~1000 tokens for
|
||||
// cyrillic: 3000 chars / 3) must split into two chunks, each kept whole.
|
||||
// Two paragraphs that together exceed the OUTPUT-token budget (each ~1156 out-tokens for
|
||||
// cyrillic: 3000 chars × 0.3852) must split into two draft chunks, each kept whole.
|
||||
p1 := strings.Repeat("а", 3000)
|
||||
p2 := strings.Repeat("б", 3000)
|
||||
got := SplitChunks([]string{p1 + "\n\n" + p2})
|
||||
got := SplitChunks([]string{p1 + "\n\n" + p2}, testSeg())
|
||||
want := []Chunk{
|
||||
{Chapter: 1, ChunkIdx: 0, Text: p1},
|
||||
{Chapter: 1, ChunkIdx: 1, Text: p2},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||
t.Fatalf("packing = %d chunks, want 2 whole paragraphs; got %+v", len(got), summarize(got))
|
||||
}
|
||||
// WS2: the two ~1156-out chunks (2311 together ≤ 3200) group into ONE edit unit.
|
||||
if got[0].EditUnitID != got[1].EditUnitID {
|
||||
t.Fatalf("two sub-ceiling draft chunks must share an edit unit, got ids %d and %d", got[0].EditUnitID, got[1].EditUnitID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitChunksPacksSmallParagraphsTogether(t *testing.T) {
|
||||
got := SplitChunks([]string{"Абзац один.\n\nАбзац два.\n\nАбзац три."})
|
||||
got := SplitChunks([]string{"Абзац один.\n\nАбзац два.\n\nАбзац три."}, testSeg())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("small paragraphs must pack into one chunk, got %d: %+v", len(got), summarize(got))
|
||||
}
|
||||
|
|
@ -54,17 +83,17 @@ func TestSplitChunksPacksSmallParagraphsTogether(t *testing.T) {
|
|||
}
|
||||
|
||||
func TestSplitChunksOversizeParagraphSplitsAtSentences(t *testing.T) {
|
||||
// One paragraph far over the token target, built of many whole sentences, must
|
||||
// One paragraph far over the output budget, built of many whole sentences, must
|
||||
// split into >1 chunk AND never cut a sentence: every original sentence must
|
||||
// survive intact and in order across the chunks.
|
||||
sentence := strings.Repeat("слово ", 40) + "конец." // ~80 tokens each
|
||||
sentence := strings.Repeat("слово ", 40) + "конец." // ~79 out-tokens each
|
||||
var sb strings.Builder
|
||||
for i := 0; i < 40; i++ { // ~3200 tokens total ≫ 1500
|
||||
for i := 0; i < 40; i++ { // ~3160 out-tokens total ≫ 1797
|
||||
sb.WriteString(sentence)
|
||||
sb.WriteByte(' ')
|
||||
}
|
||||
para := strings.TrimSpace(sb.String())
|
||||
got := SplitChunks([]string{para})
|
||||
got := SplitChunks([]string{para}, testSeg())
|
||||
if len(got) < 2 {
|
||||
t.Fatalf("oversize paragraph must split into >1 chunk, got %d", len(got))
|
||||
}
|
||||
|
|
@ -79,41 +108,43 @@ func TestSplitChunksOversizeParagraphSplitsAtSentences(t *testing.T) {
|
|||
}
|
||||
|
||||
func TestSplitChunksShortLinesPackByTrueTokenBudget(t *testing.T) {
|
||||
// Many short dialogue paragraphs must pack by TRUE token content, not by summing
|
||||
// the per-unit 16-token floor. 150 lines of 「はい。」 are ~450 true tokens (cjk=300,
|
||||
// other=450 → 300+150), so they belong in ONE chunk. Summing the floor (150×16)
|
||||
// would wrongly flush into ~2 sub-500-char chunks the coverage gate then skips.
|
||||
// Many short dialogue paragraphs must pack by TRUE output-token content (est_out on the
|
||||
// joined text), not by summing a per-unit floor. 150 lines of 「はい。」 are ~533 out-tokens,
|
||||
// so they belong in ONE chunk (a per-unit floor would wrongly flush into sub-500-char pieces).
|
||||
var paras []string
|
||||
for i := 0; i < 150; i++ {
|
||||
paras = append(paras, "「はい。」")
|
||||
}
|
||||
got := SplitChunks([]string{strings.Join(paras, "\n\n")})
|
||||
got := SplitChunks([]string{strings.Join(paras, "\n\n")}, testSeg())
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("short lines must pack by true token budget into 1 chunk, got %d chunks", len(got))
|
||||
}
|
||||
if EstimateTokens(got[0].Text) > targetChunkTokens {
|
||||
t.Fatalf("packed chunk overshoots the budget: %d tokens", EstimateTokens(got[0].Text))
|
||||
if got[0].EstOut > testSeg().DraftBudgetOut {
|
||||
t.Fatalf("packed chunk overshoots the budget: est_out %.1f", got[0].EstOut)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitChunksOversizeSentenceIsOwnChunk(t *testing.T) {
|
||||
// A single sentence (no internal terminator) larger than the budget can only be
|
||||
// its own chunk — never split.
|
||||
big := strings.Repeat("ы", 6000) // ~2000 tokens, one un-terminated run
|
||||
got := SplitChunks([]string{big})
|
||||
// A single sentence (no internal terminator) larger than the budget can only be its own
|
||||
// chunk — never split — and is flagged OversizedSentence (passthrough, WS2 §2б).
|
||||
big := strings.Repeat("ы", 6000) // ~2311 out-tokens, one un-terminated run
|
||||
got := SplitChunks([]string{big}, testSeg())
|
||||
if len(got) != 1 || got[0].Text != big {
|
||||
t.Fatalf("an oversize sentence must be one un-split chunk, got %+v", summarize(got))
|
||||
}
|
||||
if !got[0].OversizedSentence {
|
||||
t.Fatalf("a single over-budget sentence must be flagged OversizedSentence")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSplitChunksDropsEmpties(t *testing.T) {
|
||||
// A blank/whitespace-only chapter vanishes and does NOT consume a chapter number.
|
||||
got := SplitChunks([]string{"A", " \n\n ", "B"})
|
||||
got := SplitChunks([]string{"A", " \n\n ", "B"}, testSeg())
|
||||
want := []Chunk{
|
||||
{Chapter: 1, ChunkIdx: 0, Text: "A"},
|
||||
{Chapter: 2, ChunkIdx: 0, Text: "B"},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||
t.Fatalf("empties = %+v, want %+v", got, want)
|
||||
}
|
||||
}
|
||||
|
|
@ -123,7 +154,7 @@ func TestSplitChunksDeterministic(t *testing.T) {
|
|||
"Абзац.\n\nЕщё абзац.",
|
||||
strings.Repeat("слово ", 400) + "конец.",
|
||||
}
|
||||
a, b := SplitChunks(chapters), SplitChunks(chapters)
|
||||
a, b := SplitChunks(chapters, testSeg()), SplitChunks(chapters, testSeg())
|
||||
if !reflect.DeepEqual(a, b) {
|
||||
t.Fatal("SplitChunks must be deterministic")
|
||||
}
|
||||
|
|
@ -133,7 +164,7 @@ func TestSplitChunksNeverSplitsAbbreviation(t *testing.T) {
|
|||
// Two long en paragraphs each ending mid-flow with "Mr. Smith" must not chunk
|
||||
// between "Mr." and "Smith" — the abbreviation guard keeps the sentence whole.
|
||||
p := strings.Repeat("The road went on and on. ", 100) + "It was Mr. Smith who arrived."
|
||||
got := SplitChunks([]string{p})
|
||||
got := SplitChunks([]string{p}, testSeg())
|
||||
for _, c := range got {
|
||||
if strings.HasSuffix(strings.TrimSpace(c.Text), "Mr.") {
|
||||
t.Fatalf("chunk ended at an abbreviation 'Mr.' — a sentence was cut: %q", c.Text)
|
||||
|
|
@ -141,6 +172,32 @@ func TestSplitChunksNeverSplitsAbbreviation(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestSplitChunksEditUnitGroupsWholeChunks pins the WS2 decoupling: a chapter whose draft chunks
|
||||
// together exceed EditCeilingOut is grouped into >1 edit unit at DRAFT-CHUNK boundaries (never
|
||||
// splitting a chunk), so every chunk of a unit reconstructs the unit's draft losslessly.
|
||||
func TestSplitChunksEditUnitGroupsWholeChunks(t *testing.T) {
|
||||
// Three ~1156-out paragraphs (0.3852×3000): draft chunks {p1},{p2},{p3} (each ~1156, two
|
||||
// together 2311 ≤ 1797? no — 2311>1797 so each is its own draft chunk). Edit ceiling 3200:
|
||||
// group whole chunks → {p1,p2}=2311 ≤3200, +p3=3467>3200 → unit0={p1,p2}, unit1={p3}.
|
||||
p := strings.Repeat("а", 3000)
|
||||
q := strings.Repeat("б", 3000)
|
||||
r := strings.Repeat("в", 3000)
|
||||
got := SplitChunks([]string{p + "\n\n" + q + "\n\n" + r}, testSeg())
|
||||
if len(got) != 3 {
|
||||
t.Fatalf("want 3 draft chunks, got %d", len(got))
|
||||
}
|
||||
ids := []int{got[0].EditUnitID, got[1].EditUnitID, got[2].EditUnitID}
|
||||
if ids[0] != ids[1] || ids[1] == ids[2] {
|
||||
t.Fatalf("edit-unit grouping = %v, want first two together and the third separate", ids)
|
||||
}
|
||||
// Every draft chunk belongs to exactly one unit; a unit's chunks are contiguous.
|
||||
for i := 1; i < len(got); i++ {
|
||||
if got[i].EditUnitID < got[i-1].EditUnitID {
|
||||
t.Fatalf("edit unit ids must be monotone, got %v", ids)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- source sentence splitter (zh/ja/en) ---------------------------------------
|
||||
|
||||
func TestSplitSourceSentencesTilesLosslessly(t *testing.T) {
|
||||
|
|
@ -262,7 +319,7 @@ func FuzzSplitChunksPreservesContent(f *testing.F) {
|
|||
norm := NormalizeSource(s)
|
||||
want := countNonSpace(norm)
|
||||
got := 0
|
||||
for _, c := range SplitChunks([]string{norm}) {
|
||||
for _, c := range SplitChunks([]string{norm}, testSeg()) {
|
||||
got += countNonSpace(c.Text)
|
||||
}
|
||||
if got != want {
|
||||
|
|
|
|||
|
|
@ -226,7 +226,7 @@ func TestIngestEPUBRubyChapterMatchesDenseNumbering(t *testing.T) {
|
|||
if len(doc.Ruby) != 1 || doc.Ruby[0].Chapter != 2 {
|
||||
t.Fatalf("ruby dense chapter = %#v, want chapter 2", doc.Ruby)
|
||||
}
|
||||
chunks := SplitChunks(doc.Chapters)
|
||||
chunks := SplitChunks(doc.Chapters, testSeg())
|
||||
var rubyChunkChapter int
|
||||
for _, c := range chunks {
|
||||
if strings.Contains(c.Text, "朱雀") {
|
||||
|
|
|
|||
|
|
@ -129,7 +129,15 @@ type MemoryBank struct {
|
|||
entries []memoryEntry
|
||||
ac *ahoCorasick
|
||||
keyOwners map[string][]int // normalized key → indices of entries that contributed it
|
||||
version string // memoryVersion(): hash of frozen APPROVED rows + algo versions (F1/D8)
|
||||
// enrichedVersion folds ALL foldable rows incl Source:mined — the memory version of the EDIT
|
||||
// wave (snapshot_W2). baseVersion EXCLUDES Source:mined — the memory version of the DRAFT wave
|
||||
// (snapshot_W1), so a W1.5 bank-enrichment (adding mined rows) moves ONLY snapshot_W2, keeping
|
||||
// W1 checkpoints valid (WS1 §1в, «переоплата ОДНА»). The two are DOMAIN-SEPARATED (a "base-excl-
|
||||
// mined" tag in the base fold), so base ≠ enriched for EVERY row set — even a mined-free book (a
|
||||
// W1 job must never content-address to a W2 checkpoint). The load-bearing invariant is not
|
||||
// equality but STABILITY: base stays fixed as mined rows are added; only enriched moves.
|
||||
enrichedVersion string
|
||||
baseVersion string
|
||||
}
|
||||
|
||||
// pickedEntry is one selected record for a chunk with its firing key and disposition.
|
||||
|
|
@ -170,7 +178,13 @@ type memorySelection struct {
|
|||
activeIDs map[string]injectionDisposition
|
||||
}
|
||||
|
||||
func (b *MemoryBank) Version() string { return b.version }
|
||||
// Version is the ENRICHED memory version (all approved rows incl mined) — the edit-wave / whole-book
|
||||
// memory component. BaseVersion excludes Source:mined (the draft-wave component). They are
|
||||
// DOMAIN-SEPARATED — base ≠ enriched for ANY row set, including a mined-free book (never assert
|
||||
// equality) — but base stays STABLE when a W1.5 pass adds mined rows, which is what keeps W1
|
||||
// checkpoints valid («переоплата ОДНА»).
|
||||
func (b *MemoryBank) Version() string { return b.enrichedVersion }
|
||||
func (b *MemoryBank) BaseVersion() string { return b.baseVersion }
|
||||
|
||||
// materializeMemory builds a MemoryBank from the book's stored glossary rows (ORDER
|
||||
// BY-stable — GlossaryForBook). Pure and deterministic. It computes memoryVersion as a
|
||||
|
|
@ -184,11 +198,11 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
|
|||
|
||||
for _, row := range rows {
|
||||
e := memoryEntry{
|
||||
id: row.Src + "\x1f" + row.Sense + "\x1f" + strconv.Itoa(row.SinceCh) + "\x1f" + strconv.Itoa(row.UntilCh),
|
||||
src: row.Src,
|
||||
dst: row.Dst,
|
||||
status: row.Status,
|
||||
sense: row.Sense,
|
||||
id: row.Src + "\x1f" + row.Sense + "\x1f" + strconv.Itoa(row.SinceCh) + "\x1f" + strconv.Itoa(row.UntilCh),
|
||||
src: row.Src,
|
||||
dst: row.Dst,
|
||||
status: row.Status,
|
||||
sense: row.Sense,
|
||||
sinceCh: row.SinceCh,
|
||||
untilCh: row.UntilCh,
|
||||
allowShort: row.AllowShort,
|
||||
|
|
@ -240,7 +254,8 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
|
|||
}
|
||||
sort.Strings(allKeys) // deterministic automaton construction
|
||||
b.ac = buildAC(allKeys)
|
||||
b.version = computeMemoryVersion(rows, gateOn)
|
||||
b.enrichedVersion = computeMemoryVersion(rows, gateOn) // all approved incl mined (W2)
|
||||
b.baseVersion = computeMemoryVersionScoped(rows, gateOn, true) // excl Source:mined (W1)
|
||||
return b
|
||||
}
|
||||
|
||||
|
|
@ -256,14 +271,31 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
|
|||
// glossary.id autoincrement is deliberately EXCLUDED (fresh each replace); only content
|
||||
// columns are hashed.
|
||||
func computeMemoryVersion(rows []store.GlossaryEntry, gateOn bool) string {
|
||||
return computeMemoryVersionScoped(rows, gateOn, false)
|
||||
}
|
||||
|
||||
// computeMemoryVersionScoped is computeMemoryVersion with an explicit Source-scope: excludeMined
|
||||
// drops every Source:"mined" row from the fold (WS1 §1в base-bank-version). The DRAFT-wave snapshot
|
||||
// (snapshot_W1) folds base (excludeMined=true) so a W1.5 pass that ADDS mined rows moves ONLY the
|
||||
// enriched (edit-wave) version — keeping W1 checkpoints valid, «переоплата ОДНА». Deterministic (a
|
||||
// Source filter over the same ORDER BY-stable rows). excludeMined=false is BYTE-IDENTICAL to the
|
||||
// pre-split fold (no extra field), so the enriched hash / existing snapshots do not move; the
|
||||
// excludeMined=true variant adds a domain separator so base and enriched never collide.
|
||||
func computeMemoryVersionScoped(rows []store.GlossaryEntry, gateOn, excludeMined bool) string {
|
||||
h := sha256.New()
|
||||
h.Write([]byte("tm-memory-v2\x00"))
|
||||
h.Write([]byte(memoryNormVersion + "\x00" + memoryMatchVersion + "\x00"))
|
||||
h.Write([]byte("gate:" + strconv.FormatBool(gateOn) + "\x00"))
|
||||
if excludeMined {
|
||||
h.Write([]byte("base-excl-mined\x00")) // domain separator: base ≠ enriched even over identical rows
|
||||
}
|
||||
for _, r := range rows {
|
||||
if r.Status != "approved" && !gateOn {
|
||||
continue
|
||||
}
|
||||
if excludeMined && r.Source == "mined" {
|
||||
continue
|
||||
}
|
||||
// A fixed, length-prefixed field layout so no content can forge a boundary.
|
||||
writeField(h, r.Status)
|
||||
writeField(h, r.Src)
|
||||
|
|
@ -492,39 +524,53 @@ func renderGlossaryBlock(injected []pickedEntry) string {
|
|||
return glossaryBlockHeader + "\n" + strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// editorConstraintHeader introduces the editor's dst-constraint block. It gives the editor the
|
||||
// approved TARGET renderings as consistency constraints — it must normalise any divergent rendering
|
||||
// in the draft to these canonical forms (inflecting for context) and touch nothing else.
|
||||
const editorConstraintHeader = "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике эти сущности должны быть переданы ИМЕННО этими формами — приводи к ним любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):"
|
||||
// renderFormatVersion versions the FORMAT of the role-injection renderers whose output is NOT
|
||||
// captured by memoryMatchVersion (a scope/matcher-ALGORITHM version, not a render-layout version):
|
||||
// today the editor constraint block's src→dst layout (WS2 §2в) and, later, the gender-constraint
|
||||
// render (WS5). A format change shifts the injected bytes → the wire, so it must be a loud
|
||||
// --resnapshot; folding it as a dedicated snapshot component (snapshot.go) keeps that loudness a
|
||||
// MECHANISM, not discipline — memoryMatchVersion (const memory.go) would not move on a render edit.
|
||||
const renderFormatVersion = "renderfmt-v1-editor-src2dst"
|
||||
|
||||
// renderEditorConstraintBlock serializes the selected records' CONFIRMED dst renderings into the
|
||||
// editor's injection. Unlike renderGlossaryBlock it emits the TARGET forms only — bare canonical
|
||||
// Russian, not "src → dst". It is CONFIRMED-only on purpose: an AMBIGUOUS record is a candidate the
|
||||
// translator MAY have legitimately rejected, so forcing the editor to rewrite the draft toward it
|
||||
// would corrupt a correct translation (mirrors the gate's CONFIRMED-only discipline, external-review
|
||||
// #1). NOTE (D30.1 supersede of D1): the editor is now BILINGUAL — editor.md feeds it the source, so
|
||||
// the earlier "monolingual editor never sees the source" rationale for target-forms-only is stale
|
||||
// (the CONFIRMED-only discipline above is the LIVE reason and is unchanged here). Whether a bilingual
|
||||
// editor should receive the src→dst mapping (like the translator) so it can bind each canonical form
|
||||
// to its source term, rather than bare Russian, is an OPEN design question the D30.1 flip left
|
||||
// unclosed (flagged to the orchestrator; behaviour deliberately unchanged in this pack). Deduped by
|
||||
// dst (an alias and its main entry share a rendering), in the budget priority order fixed by Select.
|
||||
// Returns "" when nothing CONFIRMED renders → the editor gets NO injection (the plain draft-only
|
||||
// editorConstraintHeader introduces the editor's canonical-constraint block. It gives the BILINGUAL
|
||||
// editor (D30.1) the approved src→dst bindings as consistency constraints — it must render each
|
||||
// listed source term with EXACTLY the paired canonical form (inflecting for context) and touch
|
||||
// nothing else.
|
||||
const editorConstraintHeader = "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):"
|
||||
|
||||
// renderEditorConstraintBlock serializes the selected records' CONFIRMED renderings into the
|
||||
// editor's injection as a src→dst MAPPING (WS2 §2в resolves the D30.1-open question): "源термин →
|
||||
// «dst»", like the translator block, NOT bare canonical Russian. Rationale (the single load-bearing
|
||||
// one, ревью-1 F4): the editor is BILINGUAL (editor.md feeds it the source), and binding the canon
|
||||
// to its SOURCE term disambiguates HOMONYMIC dst (one Russian surface for two entities) — bare
|
||||
// forms cannot. It stays CONFIRMED-only on purpose: an AMBIGUOUS record is a candidate the
|
||||
// translator MAY have legitimately rejected, so forcing the editor to rewrite toward it would
|
||||
// corrupt a correct translation (mirrors the gate's CONFIRMED-only discipline, external-review #1);
|
||||
// AMBIGUOUS/mined-draft rows never enter here. Deduped by (src,dst) — an alias and its main entry
|
||||
// share the pair — which only WIDENS the list (harmless), in the budget priority order fixed by
|
||||
// Select. Returns "" when nothing CONFIRMED renders → the editor gets NO injection (plain draft-only
|
||||
// layout). The block is a subset of the already budget-limited memSel.injected, so it needs no
|
||||
// separate token budget.
|
||||
// separate token budget. The src→dst FORMAT is snapshot-folded via renderFormatVersion (a format
|
||||
// edit is a loud --resnapshot); the injection is a message, so it also enters request_hash
|
||||
// directly (no silent false-hit).
|
||||
func renderEditorConstraintBlock(injected []pickedEntry) string {
|
||||
var lines []string
|
||||
seen := map[string]bool{}
|
||||
seen := map[[2]string]bool{}
|
||||
for _, p := range injected {
|
||||
if p.disp != memConfirmed {
|
||||
continue
|
||||
}
|
||||
src := strings.TrimSpace(p.entry.src)
|
||||
dst := strings.TrimSpace(p.entry.dst)
|
||||
if dst == "" || seen[dst] {
|
||||
if dst == "" {
|
||||
continue
|
||||
}
|
||||
seen[dst] = true
|
||||
lines = append(lines, "- «"+dst+"»")
|
||||
key := [2]string{src, dst}
|
||||
if seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
lines = append(lines, "- "+src+" → «"+dst+"»")
|
||||
}
|
||||
if len(lines) == 0 {
|
||||
return ""
|
||||
|
|
|
|||
|
|
@ -529,36 +529,45 @@ func TestInjectivityCollision(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestRenderEditorConstraintBlock covers D1: the monolingual editor's injection is the
|
||||
// CONFIRMED dst forms ONLY (target constraints), deduped, with no source or "src → dst".
|
||||
// Reverting the CONFIRMED-only guard leaks an AMBIGUOUS candidate and fails here.
|
||||
// TestRenderEditorConstraintBlock covers WS2 §2в (D30.1 open-question resolved): the BILINGUAL
|
||||
// editor's injection is a CONFIRMED src→dst MAPPING (binding each canon to its source term, so a
|
||||
// homonymic dst stays distinguishable), NOT bare target forms. It stays CONFIRMED-only (an
|
||||
// AMBIGUOUS candidate must not force a rewrite); dedup is by (src,dst), which only widens the list.
|
||||
func TestRenderEditorConstraintBlock(t *testing.T) {
|
||||
suzuki := &memoryEntry{src: "鈴木", dst: "Судзуки", status: "approved"}
|
||||
cand := &memoryEntry{src: "小D", dst: "Малыш Дэ", status: "auto"} // AMBIGUOUS
|
||||
tanaka := &memoryEntry{src: "田中", dst: "Танака", status: "approved"}
|
||||
tanakaKana := &memoryEntry{src: "たなか", dst: "Танака", status: "approved"} // same dst → deduped
|
||||
tanakaDup := &memoryEntry{src: "田中", dst: "Танака", status: "approved"} // same (src,dst) → deduped
|
||||
// A distinct source term that renders to the SAME Russian surface — the homonym case the
|
||||
// src→dst binding exists for: both lines survive (widening), the editor can tell them apart.
|
||||
homonym := &memoryEntry{src: "済", dst: "Танака", status: "approved"}
|
||||
sel := []pickedEntry{
|
||||
{entry: suzuki, via: "鈴木", disp: memConfirmed},
|
||||
{entry: cand, via: "小d", disp: memAmbiguous},
|
||||
{entry: tanaka, via: "田中", disp: memConfirmed},
|
||||
{entry: tanakaKana, via: "たなか", disp: memConfirmed},
|
||||
{entry: tanakaDup, via: "田中", disp: memConfirmed},
|
||||
{entry: homonym, via: "済", disp: memConfirmed},
|
||||
}
|
||||
block := renderEditorConstraintBlock(sel)
|
||||
|
||||
if !strings.Contains(block, "«Судзуки»") || !strings.Contains(block, "«Танака»") {
|
||||
t.Errorf("confirmed dst forms missing: %q", block)
|
||||
// src→dst mapping present under the editor's own header.
|
||||
if !strings.Contains(block, "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ") {
|
||||
t.Errorf("editor header missing: %q", block)
|
||||
}
|
||||
if !strings.Contains(block, "鈴木 → «Судзуки»") || !strings.Contains(block, "田中 → «Танака»") {
|
||||
t.Errorf("src→dst mapping missing: %q", block)
|
||||
}
|
||||
// AMBIGUOUS excluded — the editor must not be forced toward an unverified rendering.
|
||||
if strings.Contains(block, "Малыш Дэ") {
|
||||
if strings.Contains(block, "Малыш Дэ") || strings.Contains(block, "小D") {
|
||||
t.Errorf("AMBIGUOUS candidate leaked into the editor constraints: %q", block)
|
||||
}
|
||||
// dst-only: no source key, no arrow (the editor is monolingual).
|
||||
if strings.Contains(block, "→") || strings.Contains(block, "鈴木") || strings.Contains(block, "ГЛОССАРИЙ") {
|
||||
t.Errorf("editor block must be target-only, no src→dst: %q", block)
|
||||
// Deduped by (src,dst): 田中→Танака appears once despite the duplicate entry.
|
||||
if n := strings.Count(block, "田中 → «Танака»"); n != 1 {
|
||||
t.Errorf("(src,dst) not deduped: 田中→Танака appears %d times: %q", n, block)
|
||||
}
|
||||
// Deduped by dst (Танака appears once despite two source keys).
|
||||
if n := strings.Count(block, "«Танака»"); n != 1 {
|
||||
t.Errorf("dst not deduped: «Танака» appears %d times: %q", n, block)
|
||||
// Homonym widening: a DIFFERENT source with the same dst is NOT collapsed — both lines survive.
|
||||
if !strings.Contains(block, "済 → «Танака»") {
|
||||
t.Errorf("homonymic dst must stay distinguishable by source term: %q", block)
|
||||
}
|
||||
// An all-AMBIGUOUS (or empty) selection yields no block at all.
|
||||
if got := renderEditorConstraintBlock([]pickedEntry{{entry: cand, disp: memAmbiguous}}); got != "" {
|
||||
|
|
|
|||
38
backend/internal/pipeline/memory_wave_test.go
Normal file
38
backend/internal/pipeline/memory_wave_test.go
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"textmachine/backend/internal/store"
|
||||
)
|
||||
|
||||
// TestBaseEnrichedMemoryVersionSplit pins the WS1 §1в per-wave bank-version mechanism: the BASE
|
||||
// version (draft wave / snapshot_W1) excludes Source:"mined" rows, so adding a mined approved row
|
||||
// after W1 moves ONLY the ENRICHED version (edit wave / snapshot_W2) — keeping W1 checkpoints valid
|
||||
// («переоплата ОДНА»). A broken split (base folding mined) would move snapshot_W1 and re-bill W1.
|
||||
func TestBaseEnrichedMemoryVersionSplit(t *testing.T) {
|
||||
seedRows := []store.GlossaryEntry{
|
||||
{Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"},
|
||||
{Src: "古月", Dst: "Гу Юэ", Status: "approved", Source: "ruby"},
|
||||
}
|
||||
base0 := materializeMemory(seedRows, false)
|
||||
// Same rows PLUS a mined approved row (as a W1.5 pass would add).
|
||||
enrichedRows := append([]store.GlossaryEntry{}, seedRows...)
|
||||
enrichedRows = append(enrichedRows, store.GlossaryEntry{Src: "蛊", Dst: "гу", Status: "approved", Source: "mined"})
|
||||
bank1 := materializeMemory(enrichedRows, false)
|
||||
|
||||
// 1) Base version is UNCHANGED by the mined addition (draft wave stays valid).
|
||||
if base0.BaseVersion() != bank1.BaseVersion() {
|
||||
t.Fatalf("base version moved on a mined-row addition — snapshot_W1 would re-bill W1:\n before %s\n after %s",
|
||||
base0.BaseVersion(), bank1.BaseVersion())
|
||||
}
|
||||
// 2) Enriched version DID move (edit wave sees the new mined canon).
|
||||
if base0.Version() == bank1.Version() {
|
||||
t.Fatalf("enriched version did NOT move on a mined-row addition — the edit wave would miss the mined canon")
|
||||
}
|
||||
// 3) Base ≠ enriched even before any mined row (domain-separated), so a W1 job can never
|
||||
// accidentally content-address to a W2 checkpoint.
|
||||
if base0.BaseVersion() == base0.Version() {
|
||||
t.Fatalf("base and enriched versions collide over identical rows — they must be domain-separated")
|
||||
}
|
||||
}
|
||||
97
backend/internal/pipeline/ratelimit.go
Normal file
97
backend/internal/pipeline/ratelimit.go
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ratelimit.go: the per-model rate-guard for the parallel waves (WS1 §1б, ревью-2 F4). The per-CALL
|
||||
// 429/Retry-After backoff already lives in the transport adapter (httpllm.go); the gap the waves
|
||||
// open is N-CONCURRENCY against a rate-limited model — mistral-large-latest fails ~48% of calls under
|
||||
// parallelism (a token-bucket / concurrency cap, tier-dependent — 00-provider-quirks §Транспорт),
|
||||
// while grok is 0%. A model caps its own wave concurrency (max_concurrency) and, optionally, paces
|
||||
// call STARTS (min_interval). This is a TRANSPORT axis: it never touches the request bytes, so it is
|
||||
// wire-neutral and NOT snapshot-folded. Guards are built once in W0, read-only in the waves; each
|
||||
// guard's internals (a buffered-channel semaphore + a mutex-protected pacer) are concurrency-safe,
|
||||
// so acquiring from N wave goroutines is race-clean.
|
||||
|
||||
// rateGuard limits concurrent calls to one model (+ optional pacing). The zero/nil guard is a no-op
|
||||
// (unlimited), so an unconfigured model — or a nil map entry — costs nothing.
|
||||
type rateGuard struct {
|
||||
sem chan struct{} // buffered to max_concurrency; nil = unlimited
|
||||
minInterval time.Duration // 0 = no pacing
|
||||
mu sync.Mutex // guards last
|
||||
last time.Time // start time of the most recent acquired call (pacer)
|
||||
}
|
||||
|
||||
// newRateGuard builds a guard. maxConcurrency ≤ 0 → unlimited concurrency; minInterval ≤ 0 → no
|
||||
// pacing. Both off → newRateGuard returns a guard that acquire treats as a pure no-op.
|
||||
func newRateGuard(maxConcurrency int, minInterval time.Duration) *rateGuard {
|
||||
g := &rateGuard{minInterval: minInterval}
|
||||
if maxConcurrency > 0 {
|
||||
g.sem = make(chan struct{}, maxConcurrency)
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
// acquire blocks until a concurrency slot is free and, if pacing is on, until the min-interval since
|
||||
// the last acquired call has elapsed — both respecting ctx cancellation. It returns a release func
|
||||
// the caller MUST defer (a no-op when acquire failed or the guard is unlimited). A nil guard (no
|
||||
// config) acquires instantly. The pacer holds the mutex only to read/stamp `last`; the wait itself
|
||||
// is outside the lock, so a paced model still admits one caller per interval without serializing the
|
||||
// lock for the whole wait.
|
||||
func (g *rateGuard) acquire(ctx context.Context) (func(), error) {
|
||||
if g == nil {
|
||||
return func() {}, nil
|
||||
}
|
||||
if g.sem != nil {
|
||||
select {
|
||||
case g.sem <- struct{}{}:
|
||||
case <-ctx.Done():
|
||||
return func() {}, ctx.Err()
|
||||
}
|
||||
}
|
||||
release := func() {
|
||||
if g.sem != nil {
|
||||
<-g.sem
|
||||
}
|
||||
}
|
||||
if g.minInterval > 0 {
|
||||
for {
|
||||
g.mu.Lock()
|
||||
wait := time.Until(g.last.Add(g.minInterval))
|
||||
if wait <= 0 {
|
||||
g.last = time.Now()
|
||||
g.mu.Unlock()
|
||||
break
|
||||
}
|
||||
g.mu.Unlock()
|
||||
select {
|
||||
case <-time.After(wait):
|
||||
// re-check: another caller may have taken the slot while we slept.
|
||||
case <-ctx.Done():
|
||||
release()
|
||||
return func() {}, ctx.Err()
|
||||
}
|
||||
}
|
||||
}
|
||||
return release, nil
|
||||
}
|
||||
|
||||
// buildRateGuards constructs a guard for every reachable model in W0 (read-only in the waves).
|
||||
// A model with no rate_limit config gets an unlimited (no-op) guard, so rateGuard(model) is always
|
||||
// non-nil and lookup-safe.
|
||||
func (r *Runner) buildRateGuards() {
|
||||
r.rateGuards = make(map[string]*rateGuard, len(r.Models.Models))
|
||||
for _, m := range r.reachableModels() {
|
||||
rl := r.Models.Models[m].RateLimit
|
||||
r.rateGuards[m] = newRateGuard(rl.MaxConcurrency, time.Duration(rl.MinIntervalMS)*time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
// rateGuard returns the pre-built guard for a model (nil-safe: acquire treats a missing/nil guard as
|
||||
// unlimited, so a model reached before buildRateGuards ran simply is not throttled).
|
||||
func (r *Runner) rateGuard(model string) *rateGuard {
|
||||
return r.rateGuards[model]
|
||||
}
|
||||
113
backend/internal/pipeline/ratelimit_test.go
Normal file
113
backend/internal/pipeline/ratelimit_test.go
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestRateGuardLimitsConcurrency proves the semaphore REALLY bounds concurrency (WS6 precondition,
|
||||
// ревью-2 F4): with max_concurrency=2, no more than 2 goroutines are ever inside the guarded region
|
||||
// at once, even under 20 racing callers. A broken guard (missing/over-sized semaphore) trips the
|
||||
// observed-max assertion.
|
||||
func TestRateGuardLimitsConcurrency(t *testing.T) {
|
||||
const cap, workers = 2, 20
|
||||
g := newRateGuard(cap, 0)
|
||||
var inFlight, maxSeen int64
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < workers; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
rel, err := g.acquire(context.Background())
|
||||
if err != nil {
|
||||
t.Errorf("acquire: %v", err)
|
||||
return
|
||||
}
|
||||
cur := atomic.AddInt64(&inFlight, 1)
|
||||
for {
|
||||
m := atomic.LoadInt64(&maxSeen)
|
||||
if cur <= m || atomic.CompareAndSwapInt64(&maxSeen, m, cur) {
|
||||
break
|
||||
}
|
||||
}
|
||||
time.Sleep(time.Millisecond) // widen the window a concurrency bug would exploit
|
||||
atomic.AddInt64(&inFlight, -1)
|
||||
rel()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if maxSeen > cap {
|
||||
t.Fatalf("rate-guard admitted %d concurrent callers, cap is %d", maxSeen, cap)
|
||||
}
|
||||
if maxSeen == 0 {
|
||||
t.Fatalf("no caller was observed in-flight — the guard never admitted anyone")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRateGuardNilAndUnlimitedAreNoOps: a nil guard and a zero-config guard both admit instantly and
|
||||
// never block (the sequential path / an unconfigured model must cost nothing).
|
||||
func TestRateGuardNilAndUnlimitedAreNoOps(t *testing.T) {
|
||||
var nilGuard *rateGuard
|
||||
rel, err := nilGuard.acquire(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("nil guard acquire: %v", err)
|
||||
}
|
||||
rel() // must not panic
|
||||
unlimited := newRateGuard(0, 0)
|
||||
for i := 0; i < 100; i++ {
|
||||
rel, err := unlimited.acquire(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("unlimited acquire %d: %v", i, err)
|
||||
}
|
||||
rel()
|
||||
}
|
||||
}
|
||||
|
||||
// TestRateGuardHonoursContextCancel: a caller blocked on a full semaphore unblocks on ctx cancel and
|
||||
// does NOT leak a slot (the wave dispatcher must be able to abort a run cleanly).
|
||||
func TestRateGuardHonoursContextCancel(t *testing.T) {
|
||||
g := newRateGuard(1, 0)
|
||||
rel, err := g.acquire(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("first acquire: %v", err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
r, err := g.acquire(ctx) // blocks: the one slot is held
|
||||
r()
|
||||
done <- err
|
||||
}()
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != context.Canceled {
|
||||
t.Fatalf("blocked acquire on cancel = %v, want context.Canceled", err)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("acquire did not unblock on ctx cancel")
|
||||
}
|
||||
rel() // release the first; a fresh acquire must now succeed (no leaked slot)
|
||||
r2, err := g.acquire(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("acquire after release: %v", err)
|
||||
}
|
||||
r2()
|
||||
}
|
||||
|
||||
// TestRateGuardPacingSpacesStarts: with min_interval set, two serial acquires are spaced by at least
|
||||
// the interval (best-effort pacing of call STARTS).
|
||||
func TestRateGuardPacingSpacesStarts(t *testing.T) {
|
||||
g := newRateGuard(0, 30*time.Millisecond)
|
||||
start := time.Now()
|
||||
r1, _ := g.acquire(context.Background())
|
||||
r1()
|
||||
r2, _ := g.acquire(context.Background())
|
||||
r2()
|
||||
if elapsed := time.Since(start); elapsed < 30*time.Millisecond {
|
||||
t.Fatalf("paced acquires took %v, want ≥30ms between starts", elapsed)
|
||||
}
|
||||
}
|
||||
|
|
@ -43,7 +43,11 @@ import (
|
|||
// now normalized once in NormalizeSource (#3), the ASCII sentence boundary honors
|
||||
// quote-depth (#4), and <br> inside <ruby> no longer leaks a newline into the body
|
||||
// (#5) — all segmentation/ingest behaviour changes → a loud --resnapshot.
|
||||
const chunkerVersion = "chunker-v4-utf8-quotedepth"
|
||||
// Bumped v4→v5 (WS2): the packing budget moved from input-token heuristic (const 1500) to OUTPUT
|
||||
// (ru) tokens via fertility (est_out), draft chunks decoupled from coarse EDIT units (grouped whole
|
||||
// chunks to EditCeilingOut), oversized-sentence flag added. The budget itself is ALSO folded via
|
||||
// segmentationSnap (snapshot.go), so this version covers only the algorithm shape.
|
||||
const chunkerVersion = "chunker-v5-output-budget-editunit"
|
||||
|
||||
// estimatorVersion versions EstimateTokens: его выход входит в max_tokens и
|
||||
// через него в request-hash, поэтому перекалибровка весов — тоже явная
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package pipeline
|
|||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
|
||||
|
|
@ -43,6 +44,9 @@ type Runner struct {
|
|||
|
||||
clients map[string]llm.LLMClient
|
||||
templates map[string]*PromptTemplate
|
||||
// rateGuards is the per-model wave-concurrency guard set (WS1 §1б), built once in W0 and
|
||||
// read-only in the waves — a transport axis, never snapshot-folded. nil until buildRateGuards.
|
||||
rateGuards map[string]*rateGuard
|
||||
|
||||
// memory is the book's glossary FROZEN for this job (materialized once in
|
||||
// TranslateBook, after seeding, before snapshotID). Its Version() is the F1
|
||||
|
|
@ -168,14 +172,63 @@ func (r *Runner) loadTemplates() error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// segBudget resolves the pipeline's config.Segmentation into the pipeline-package SegBudget the
|
||||
// chunker consumes (WS2). Kept in the runner (not chunker.go) so the segmenter stays config-free.
|
||||
func (r *Runner) segBudget() SegBudget {
|
||||
seg := r.Pipeline.Segmentation
|
||||
return SegBudget{
|
||||
DraftBudgetOut: float64(seg.DraftBudgetOut),
|
||||
EditCeilingOut: float64(seg.EditCeilingOut),
|
||||
FertCJK: seg.Fertility.CJK,
|
||||
FertOther: seg.Fertility.Other,
|
||||
}
|
||||
}
|
||||
|
||||
// reachableModels enumerates every model a run can call: the union of the stage models and their
|
||||
// single-hop escalate_to fallbacks (stagerun.go calls r.client ONLY with model∈{st.Model, EscalateTo}).
|
||||
// This set is complete and static for a book, which is what lets W0 pre-build every client (a third
|
||||
// model axis — channel B / annotator — must extend this to stay race-free). Deterministic order.
|
||||
func (r *Runner) reachableModels() []string {
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
add := func(m string) {
|
||||
if m != "" && !seen[m] {
|
||||
seen[m] = true
|
||||
out = append(out, m)
|
||||
}
|
||||
}
|
||||
for _, st := range r.Pipeline.Stages {
|
||||
add(st.Model)
|
||||
add(st.EscalateTo)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// buildClients EAGER-constructs every reachable LLM client BEFORE any wave goroutine starts (W0).
|
||||
// After this the clients map is READ-ONLY in the waves, so r.client is lock-free and a miss is a
|
||||
// loud error, not a lazy build under a data race — closing the runner.go lazy-init race (D12) and
|
||||
// tripwiring a future un-enumerated model axis. Idempotent; BuildClient needs only the provider
|
||||
// config (not the API key), so eager-build is safe even on a key-less resume.
|
||||
func (r *Runner) buildClients() error {
|
||||
for _, m := range r.reachableModels() {
|
||||
if _, ok := r.clients[m]; ok {
|
||||
continue
|
||||
}
|
||||
c, err := BuildClient(r.Models, m, r.Log)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pipeline: eager-build client for model %q (W0): %w", m, err)
|
||||
}
|
||||
r.clients[m] = c
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// client returns the PRE-BUILT client for a model. Read-only (no lazy build, no lock): buildClients
|
||||
// constructed every reachable client in W0, so the map is only read in the waves; a miss means an
|
||||
// un-enumerated model reached the wire and is a loud error, never a silent lazy build under a race.
|
||||
func (r *Runner) client(model string) (llm.LLMClient, error) {
|
||||
if c, ok := r.clients[model]; ok {
|
||||
return c, nil
|
||||
}
|
||||
c, err := BuildClient(r.Models, model, r.Log)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r.clients[model] = c
|
||||
return c, nil
|
||||
return nil, fmt.Errorf("pipeline: no pre-built client for model %q — W0 buildClients enumerates stage models ∪ escalate_to; a model outside that set reached the wire, add it to the eager set to keep the waves race-free", model)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -89,9 +89,9 @@ func TestRunnerMemoryInjectionAndPostcheck(t *testing.T) {
|
|||
if strings.Contains(msgs[2].Content, "ГЛОССАРИЙ") {
|
||||
t.Errorf("glossary leaked into the user (ch.Text) message: %q", msgs[2].Content)
|
||||
}
|
||||
// D1: the monolingual EDITOR now gets the CONFIRMED dst forms as target constraints — its
|
||||
// OWN dst-constraint block (distinct header), NOT the translator's "src → dst" ГЛОССАРИЙ,
|
||||
// and never the source key inside the injection.
|
||||
// D30.1/WS2 §2в: the BILINGUAL editor gets the CONFIRMED canon as a src→dst MAPPING under its
|
||||
// OWN header (КАНОНИЧЕСКИЕ), binding each canonical form to its source term (homonym-safe) — NOT
|
||||
// the translator's ГЛОССАРИЙ header, and never an AMBIGUOUS row.
|
||||
editorSaw := false
|
||||
for _, b := range rec.all() {
|
||||
if !isEditBody(b) {
|
||||
|
|
@ -101,11 +101,11 @@ func TestRunnerMemoryInjectionAndPostcheck(t *testing.T) {
|
|||
if len(em) != 3 || em[1].Role != "system" {
|
||||
t.Fatalf("editor must have 3 messages (system, dst-constraints, user), got %d: %+v", len(em), em)
|
||||
}
|
||||
if !strings.Contains(em[1].Content, "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ") || !strings.Contains(em[1].Content, "Судзуки") {
|
||||
t.Errorf("editor dst-constraint block missing/wrong: %q", em[1].Content)
|
||||
if !strings.Contains(em[1].Content, "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ") || !strings.Contains(em[1].Content, "鈴木 → «Судзуки»") {
|
||||
t.Errorf("editor src→dst constraint block missing/wrong: %q", em[1].Content)
|
||||
}
|
||||
if strings.Contains(em[1].Content, "ГЛОССАРИЙ") || strings.Contains(em[1].Content, "→") || strings.Contains(em[1].Content, "鈴木") {
|
||||
t.Errorf("editor block must be dst-only (no ГЛОССАРИЙ header / arrow / source): %q", em[1].Content)
|
||||
if strings.Contains(em[1].Content, "ГЛОССАРИЙ") {
|
||||
t.Errorf("editor block must use its own КАНОНИЧЕСКИЕ header, not the translator ГЛОССАРИЙ: %q", em[1].Content)
|
||||
}
|
||||
editorSaw = true
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,15 +11,36 @@ import (
|
|||
// громкий --resnapshot (= переоплата книги, D15), никогда тихий false-hit.
|
||||
|
||||
// contextSnap freezes the context-assembly knobs (§3.8) inside the snapshot.
|
||||
// Fixed field order — it is part of a content hash.
|
||||
// Fixed field order — it is part of a content hash. WS2 dropped STMDepth/OverlapTokens
|
||||
// (dead carryover knobs — §2а), a structural change folded loudly (§8 manifest).
|
||||
type contextSnap struct {
|
||||
GlossaryInjection string `json:"glossary_injection"`
|
||||
GlossaryTokenBudget int `json:"glossary_token_budget"`
|
||||
STMDepth int `json:"stm_depth"`
|
||||
OverlapTokens int `json:"overlap_tokens"`
|
||||
CacheTTL string `json:"cache_ttl"`
|
||||
}
|
||||
|
||||
// segmentationSnap freezes the WS2 output-token chunking budget (draft/edit ceilings + per-pair
|
||||
// fertility) inside the snapshot, mirroring coverageSnap's discipline: a budget or fertility edit
|
||||
// re-chunks the book (changes chunk boundaries → the wire), so it is a loud --resnapshot (D30.9),
|
||||
// never a silent divergence. Unlike coverageSnap it is ALWAYS folded — segmentation always runs and
|
||||
// its values are wire-determining. Fixed field order — part of a content hash.
|
||||
type segmentationSnap struct {
|
||||
DraftBudgetOut int `json:"draft_budget_out"`
|
||||
EditCeilingOut int `json:"edit_ceiling_out"`
|
||||
FertilityCJK float64 `json:"fertility_cjk"`
|
||||
FertilityOther float64 `json:"fertility_other"`
|
||||
}
|
||||
|
||||
func (r *Runner) segmentationSnapshot() segmentationSnap {
|
||||
seg := r.Pipeline.Segmentation
|
||||
return segmentationSnap{
|
||||
DraftBudgetOut: seg.DraftBudgetOut,
|
||||
EditCeilingOut: seg.EditCeilingOut,
|
||||
FertilityCJK: seg.Fertility.CJK,
|
||||
FertilityOther: seg.Fertility.Other,
|
||||
}
|
||||
}
|
||||
|
||||
// coverageSnap freezes the excision coverage-gate config inside the snapshot, so
|
||||
// enabling the gate OR tuning its thresholds is a loud --resnapshot, never a silent
|
||||
// mismatch between an old checkpoint's stored disposition and a changed gate. HONEST
|
||||
|
|
@ -182,6 +203,14 @@ func (r *Runner) snapshotID() (id, payload string, err error) {
|
|||
// ваем ДО инъекции, чтобы смена budget'а инъекции не промахнулась мимо
|
||||
// resnapshot-гейта (D5.2). Фикс-порядок полей — это content-hash.
|
||||
ContextAssembly contextSnap `json:"context_assembly"`
|
||||
// Segmentation — the WS2 output-token chunk/edit-unit budget + fertility (segmentationSnap).
|
||||
// It changes chunk boundaries → the wire, so a budget/fertility edit is a loud --resnapshot.
|
||||
Segmentation segmentationSnap `json:"segmentation"`
|
||||
// RenderFormatVersion versions the FORMAT of the role-injection renderers not captured by
|
||||
// memoryMatchVersion (a scope/matcher-algorithm version): the editor constraint block's
|
||||
// src→dst layout (WS2 §2в) and the gender-constraint render (WS5). A format change shifts the
|
||||
// injected bytes → the wire, so folding it separately keeps a flip a loud --resnapshot.
|
||||
RenderFormatVersion string `json:"render_format_version"`
|
||||
// MemoryVersion — content-hash детерминированно материализованной инъекти-
|
||||
// руемой памяти (approved-глоссарий+резюме+series-bible). Не bump-счётчик
|
||||
// (D8): забыть пересчитать нельзя, а забытый bump переоткрыл бы класс тихой
|
||||
|
|
@ -222,15 +251,15 @@ func (r *Runner) snapshotID() (id, payload string, err error) {
|
|||
ContextAssembly: contextSnap{
|
||||
GlossaryInjection: r.Pipeline.Context.GlossaryInjection,
|
||||
GlossaryTokenBudget: r.Pipeline.Context.GlossaryTokenBudget,
|
||||
STMDepth: r.Pipeline.Context.STMDepth,
|
||||
OverlapTokens: r.Pipeline.Context.OverlapTokens,
|
||||
CacheTTL: r.Pipeline.Context.CacheTTL,
|
||||
},
|
||||
MemoryVersion: r.memoryVersion(),
|
||||
PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate,
|
||||
Coverage: r.coverageSnapshot(),
|
||||
StyleCheckVersion: cheapGateVersion,
|
||||
Sanitizer: r.sanitizerSnapshot(),
|
||||
Segmentation: r.segmentationSnapshot(),
|
||||
RenderFormatVersion: renderFormatVersion,
|
||||
MemoryVersion: r.memoryVersion(),
|
||||
PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate,
|
||||
Coverage: r.coverageSnapshot(),
|
||||
StyleCheckVersion: cheapGateVersion,
|
||||
Sanitizer: r.sanitizerSnapshot(),
|
||||
}
|
||||
for _, st := range r.Pipeline.Stages {
|
||||
ss := stageSnap{
|
||||
|
|
|
|||
|
|
@ -352,6 +352,18 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID
|
|||
}
|
||||
att.freshCall = true
|
||||
|
||||
// Per-model rate-guard (WS1 §1б): a wave-concurrency cap (+ optional pacing) around the ONE
|
||||
// expensive provider call, so a rate-limited model (mistral) does not 429-storm under N-parallel
|
||||
// waves. A no-op for an unconfigured model / the sequential path. Acquire respects ctx; on cancel
|
||||
// the reservation is released and the call surfaces as an infra failure (like a dead client).
|
||||
relGuard, err := r.rateGuard(model).acquire(ctx)
|
||||
if err != nil {
|
||||
r.releaseReservation(ctx, resv)
|
||||
r.setJobStatus(ctx, job.ID, "failed")
|
||||
return att, fmt.Errorf("pipeline: rate-guard acquire for %s/ch%d/chunk%d/%s (model %s): %w", r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, model, err)
|
||||
}
|
||||
defer relGuard()
|
||||
|
||||
// «Вызов в полёте» обязан быть виден: между этой строкой и «attempt completed»
|
||||
// могут пройти минуты (attempt_timeout × ретраи транспорта), и без неё «висит
|
||||
// провайдер» неотличимо от «умер процесс» — главная боль smoke-прогона №4.
|
||||
|
|
|
|||
|
|
@ -141,7 +141,7 @@ func (r *Runner) bookChunks() ([]Chunk, error) {
|
|||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return SplitChunks(doc.Chapters), nil
|
||||
return SplitChunks(doc.Chapters, r.segBudget()), nil
|
||||
}
|
||||
|
||||
// chunkKey identifies a chunk positionally.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -1,3 +1,5 @@
|
|||
> **⚠ СТАТУС: ЧАСТИЧНО ОТРАБОТАН — Block A + ядра ПРИНЯТЫ И ЗАЛЕНДЕНЫ (19.07, D39.13).** Сдано: WS2 ✅ (чанкер output-бюджет + edit-единицы + src→dst + golden) · WS1a/b/c-инфра ✅ (eager-clients, rate-guard, base/enriched split) · WS4-core ✅ (банкнота-парсер) · WS6-частично (конфиг-тай). **Резидуал R1–R5** (волновой ИСПОЛНИТЕЛЬ · WS3-майнер · WS4-12-точек · WS5-чекеры · арм-конфиги) — точные дизайны в `backend/PACK11_REPORT.md` §Резидуал. **Промт ОСТАЁТСЯ АКТИВНЫМ:** сессия-продолжение стартует по нему + отчёту (читать ОБА; сданное не пере-строить — оно под тестами).
|
||||
|
||||
# Промт: сессия БЭКЕНД — единый пак по плану 11 (волновой раннер · чанкер/единицы · банк W1.5 · банкнота · чекеры · армы), 2026-07-19
|
||||
|
||||
> **Место в очереди (пост-D39.12):** исследовательская программа арх-ресета (D39–D39.10) и дизайн-синтез
|
||||
|
|
|
|||
|
|
@ -4,11 +4,15 @@
|
|||
> - **Фазы:** 0 ✅ · Ф1-инфра ✅ (D20–D28; golden = инвариант №8) · арка «качество-первым» D26→D38.5 ЗАКРЫТА (хроника — `archive/PROGRESS-2026-07-10-13.md`; итог: потолок в организации, не в моделях). **АРХ-РЕСЕТ D39 (13.07):** синк-аудит «сломанного телефона» (65 находок / 0 refuted — `architecture/08-sync-audit-ledger.md`) → целевая 7-слойная архитектура + фазовый план (`architecture/09-target-architecture.md`); инвариант общности §0.1 (любая книга/пара/структура); промпт-тема консолидирована в `architecture/10-prompt-architecture.md`.
|
||||
> - **Трек A (build-now) ЗАКРЫТ:** пак-1 (D39.2, `d3f6b34`: trust-gated suppressor [терм-дрейф закрыт в КОДЕ] · export-contract нормализация · pair-сеам `prompts:{zh-ru}` fail-loud · реестр role→инъекция · target-гейты · quality-report) + пак-1.5+F1–F9 (D39.5, `dc30c7d`: `tmctl export` [manifest-join+drift-гард; полигон-экстракция впредь ТОЛЬКО через него] · fold-first санитайзер [HIGH-обход закрыт] · bounded глосс-whitelist · пример zh→ru). Адверсариал-долг 529 погашен (D39.4: 8 осей, 0 CRIT/1 HIGH/6 MED — всё в фикс-листах/ledger).
|
||||
> - **Трек B (ресёрч) ЗАЛЕНДЕН:** research/19 нарезка+когезия+контракт t/e+волны W0–W3 (D39.1; конфликт exp14↔research/18 разрешён: эмиссия≠окно≠связность) · research/20 банк-майнинг W1.5 (D39.6; детектор-лестница V-A/B/C, алиас-ярусы, banknote-v1, плагины P1–P6, §D-пре-рег; конфаунд инъекции ⇒ cold-start срез).
|
||||
> - **exp15 ЗАВЕРШЁН И ЗАЛЕНДЕН (D39.7, `fe34a16`):** «граница вредит» отозван (артефакт окна судьи, свинг +0.689 после фикса), все когезия-эффекты под шум-полом 0.126, «sequential нигде не лучше» (LOO-stable, НЕ ратифицированный зелёный свет волне), фертильность 1.20·cjk+0.39·other; $12.79/$15. **Слепое чтение ИСПОЛНЕНО (D39.8):** планка не пройдена никем, дефекты распределены по армам (читатель подтвердил floor-null; V2-брак = риг без гейтов, не carryover-рычаг) → битва качества пере-прогона = дефект-классы (时辰-юниты · числа b1 · gender-enforce · стих/аллюзии · регистр-лексикон), не нарезка. **Q4a ИСПОЛНЕН И ЗАЛЕНДЕН (D39.9, $1.32/$2):** верность НЕ покупается на translate-стадии (flash уже верен 25/0/27; pro ×3.67 не ратифицирован); смысл ломает glm-РЕДАКТОР (свап glm→mistral/deepseek-pro = приоритетный арм пере-прогона); exp14b-правила дефектны → $0-ре-аудит в exp16. **exp16 ИСПОЛНЕН И ЗАЛЕНДЕН (D39.10, $0.02/$10):** WHICH подтверждена (код-детектор V-C recall 0.97 held-out, $0) / WHAT опровергнута для ко-оккуренции → dst-канал = банкнота (идея владельца оправдана) + облако+Палладий; 9B-споттер не нужен (0.895 ≤ код); Z1 → облако; ре-аудит exp14b: ядро D38 держится, errata не нужна. Тачпойнты владельцу: карта подписи (пол) · мини-голд алиасов · precision@30 (books/gu-zhenren/exp16/). **Исследовательская программа ЗАВЕРШЕНА.** **Директива владельца (D39.11): стройка ТОЛЬКО после ДИЗАЙН-СИНТЕЗ сессии** (синтез всех находок → план реализации с верифицированными алгоритмами → ратификация → бэкенд-промт). **Очередь:** ✅ дизайн-синтез ИСПОЛНЕН, план РАТИФИЦИРОВАН (**D39.12** — `architecture/11-implementation-plan.md`, три рубежа верификации, 4 гейтнутых арма) → **бэкенд-пак ЕДИНОЙ сессией (промт ВЫДАН — `BACKEND_PLAN11_SESSION_PROMPT.md`; решение владельца)** → приёмка оркестратора → единый resnapshot (D30.9, одна переоплата; §8-манифест плана; вкл. RegressionGuard-тумблер D38.4) → пере-прогон 3–10 глав (армы редактора glm/mistral/deepseek-pro) → чтение владельца (интерим-планка 2 претензии, D35).残: POLYGON_PACKAGE4 residual-трекер (пилот/18+/echo — отложен, D36.1).
|
||||
> - **exp15 ЗАВЕРШЁН И ЗАЛЕНДЕН (D39.7, `fe34a16`):** «граница вредит» отозван (артефакт окна судьи, свинг +0.689 после фикса), все когезия-эффекты под шум-полом 0.126, «sequential нигде не лучше» (LOO-stable, НЕ ратифицированный зелёный свет волне), фертильность 1.20·cjk+0.39·other; $12.79/$15. **Слепое чтение ИСПОЛНЕНО (D39.8):** планка не пройдена никем, дефекты распределены по армам (читатель подтвердил floor-null; V2-брак = риг без гейтов, не carryover-рычаг) → битва качества пере-прогона = дефект-классы (时辰-юниты · числа b1 · gender-enforce · стих/аллюзии · регистр-лексикон), не нарезка. **Q4a ИСПОЛНЕН И ЗАЛЕНДЕН (D39.9, $1.32/$2):** верность НЕ покупается на translate-стадии (flash уже верен 25/0/27; pro ×3.67 не ратифицирован); смысл ломает glm-РЕДАКТОР (свап glm→mistral/deepseek-pro = приоритетный арм пере-прогона); exp14b-правила дефектны → $0-ре-аудит в exp16. **exp16 ИСПОЛНЕН И ЗАЛЕНДЕН (D39.10, $0.02/$10):** WHICH подтверждена (код-детектор V-C recall 0.97 held-out, $0) / WHAT опровергнута для ко-оккуренции → dst-канал = банкнота (идея владельца оправдана) + облако+Палладий; 9B-споттер не нужен (0.895 ≤ код); Z1 → облако; ре-аудит exp14b: ядро D38 держится, errata не нужна. Тачпойнты владельцу: карта подписи (пол) · мини-голд алиасов · precision@30 (books/gu-zhenren/exp16/). **Исследовательская программа ЗАВЕРШЕНА.** **Директива владельца (D39.11): стройка ТОЛЬКО после ДИЗАЙН-СИНТЕЗ сессии** (синтез всех находок → план реализации с верифицированными алгоритмами → ратификация → бэкенд-промт). **Очередь:** ✅ план РАТИФИЦИРОВАН (D39.12) → ✅ **пак-11 Block A ПРИНЯТ и залендён (D39.13:** WS2 целиком [вкл. src→dst = закрытие D30.1] + WS1a/b/c-инфра + WS4-ядро; golden re-captured**)** → **продолжение-пак R1–R5** (волновой исполнитель · WS3-майнер · WS4-12-точек · WS5-чекеры · арм-конфиги; промт `BACKEND_PLAN11_SESSION_PROMPT.md` АКТИВЕН со статус-баннером + `backend/PACK11_REPORT.md` §Резидуал) → приёмка (с погашением долга рубежа-2) → единый resnapshot (D30.9, §8-манифест) → пере-прогон 3–10 глав (армы glm/mistral/deepseek-pro) → чтение владельца (интерим-планка 2 претензии, D35).残: POLYGON_PACKAGE4 residual-трекер (пилот/18+/echo — отложен, D36.1).
|
||||
> - **Стек (не менялся с D38.5):** draft deepseek-v4-flash (thinking ON) → editor glm-5 БИЛИНГВ v3-discourse (P1a+чэнъюй+few_shot-тумблер; свап-кандидаты mistral/deepseek-pro = армы пере-прогона) → судья gemini-3.1-pro-preview; канал B Mistral+grok; 7 ключей; ~$0.85/ранобэ (D30.4).
|
||||
> - **Ждём от владельца:** запуск exp15 (бюджет ≤$15 подтверждён) · research/20 §E: мини-голд алиасов (~20–30 мин) / бюджет ≤$5 полигон-стадии / политика канона / жанр-паки / реплика ja→ru · чтение пере-прогона (после очереди) · старые висящие: планка запуска (лучше-фана/гибрид/издательский) · билингв-якорь пилота (D25.9-Q1) · контаминация пилот-корпуса (D27.4) · publishable/waiver (D25.1) · FN-bound L3 (D25.4) · юр-пакет · провенанс 12-*-доков.
|
||||
> - Архивы хроники: `archive/PROGRESS-2026-07-04-10.md` (D31) · `archive/PROGRESS-2026-07-10-13.md` (D39.6-гигиена). Записи ниже — живая эра D39.
|
||||
|
||||
## Оркестратор №7 — пак-11 Block A принят и залендён (D39.13), 19.07 (третья фаза сессии)
|
||||
|
||||
Бэкенд-сессия сдала Block A + ядра с ЧЕСТНЫМ резидуалом (мандат «резидуал > сломанный суит»): WS2 целиком (output-бюджет чанкера, edit-единицы grouping-of-whole-chunks [девиация ратифицирована: 37 единиц / 0 straddle воспроизведены, гарантия W2-реконструкции по построению], src→dst-блок редактора = закрытие D30.1, `render-format-version`, golden re-capture) · WS1a eager-clients (гонка закрыта) · WS1b rate-guard (`ratelimit.go`) · WS1c base/enriched `MemoryVersion`-split (enriched байт-идентичен — снапшот не двинулся) · WS4-core банкнота-парсер (байт-верный порт). **Приёмка — верификация исполнением:** build/vet/race сам, 27 пинов свежим `-count=1` вербозно, независимая masked-golden репликация (все категории объяснимы), построчное чтение несущих диффов. Сессионный 9-агентный чекпойнт: 3 confirmed пофикшены / 3 refuted. **Долг: мульти-агентный рубеж-2 приёмки не гонялся (лимит-режим) — гасится при приёмке продолжения (прецедент D39.2).** Резидуал R1–R5 с точными дизайнами — `backend/PACK11_REPORT.md`; промт пака АКТИВЕН со статус-баннером. Записка продолжению-сессии: стартовать по промту + отчёту, сданное не пере-строить.
|
||||
|
||||
## Оркестратор №7 — план 11 РАТИФИЦИРОВАН (D39.12), бэкенд-промт единого пака ВЫДАН (19.07, вторая половина сессии)
|
||||
|
||||
Дизайн-синтез сдал план (1066 строк) после трёх рубежей: ревью-2 оркестратора (F1–F11: 3 MAJOR соло-прохода + 2 замечания владельца [mistral rate-guard ~48% retry-fails · полы per-editor-model] + Q9-провенанс) → 7-агентный пост-фикс адверсариал сессии (5 находок: F8 ПЕРЕПИСАН — сайзинг уже от `prev`; дефолт 3200 → 37 edit-единиц; mined-write-путь [`loadGlossarySeed` хардкодит seed]; render-format-version реконсиляция; F6 усилен narrative-coref) → 3-агентный подтверждающий пас (2 резидуала закрыты). Спот-чек внесения — исполнением (36 маркеров, grep всех фиксов). **Залендено одним пакетом:** план с ревью-шапкой + `eval/design11/` (7 скриптов, 6 верификаций PASS) + quirks-запись mistral (из §12-заготовки, зона полигона — с пометкой) + D39.12 + архивация спент-промта дизайн-синтеза. **Гейтнутые армы: 4** (edit>3200 за Q2a · качество свап-арма за платным полигоном · DC-landing за $0-FP · mistral за rate-guard). Развилки §10 (14 шт., с рекомендациями) — у владельца, стройку не блокируют. **Выдан `BACKEND_PLAN11_SESSION_PROMPT.md`** — единый пак WS1–WS6 блоками A/B/C/D с чекпойнт-ревью (решение владельца: одной сессией; свободы: рефакторинг/перенос кода, временные тесты, багфиксы со скоуп-пометкой; Go-каноны; даблчек неконсистентностей; $0 к провайдерам). Очередь: бэкенд-сессия → приёмка (рубеж-2) → единый resnapshot → пере-прогон → чтение.
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
Loading…
Add table
Reference in a new issue