Land backend pack-11 block A: output-token chunker with edit units, src-to-dst editor block, eager clients, per-model rate guard, memory version split, banknote parser core
This commit is contained in:
parent
1f285250ab
commit
85f5b9e633
27 changed files with 1402 additions and 303 deletions
239
backend/PACK11_REPORT.md
Normal file
239
backend/PACK11_REPORT.md
Normal file
|
|
@ -0,0 +1,239 @@
|
||||||
|
# PACK11_REPORT — бэкенд-сессия по плану 11 (2026-07-19)
|
||||||
|
|
||||||
|
> Строил по `docs/architecture/11-implementation-plan.md` (ратифицирован D39.12). **НЕ коммитил** —
|
||||||
|
> сдача на приёмку оркестратору. Верификация исполнением: `go build ./... && go vet ./... &&
|
||||||
|
> go test -race ./...` — зелёное (логи ниже). Все несущие изменения сверены с живым HEAD-кодом.
|
||||||
|
|
||||||
|
## 0. Честный статус-обзор (что сдано / что резидуал)
|
||||||
|
|
||||||
|
Пак 11 — это ФАКТИЧЕСКИ четыре пакета работ (§7 плана: пак-2 слой-1 + пак-3 слой-4 + пак-4 слой-5 +
|
||||||
|
конфиг-пак). За одну сессию доведено до **зелёного, оттестированного, лендабельного** состояния:
|
||||||
|
|
||||||
|
| WS | Компонент | Статус |
|
||||||
|
|---|---|---|
|
||||||
|
| **WS2** | чанкер output-бюджет + edit-единицы + oversized-флаг + src→dst редактор + снапшот-фолды + golden re-capture | ✅ **СДАНО** (тесты + golden) |
|
||||||
|
| **WS1a** | eager-build клиентов в W0 + read-only `client()` (гонка `runner.go` закрыта) | ✅ **СДАНО** (`-race`) |
|
||||||
|
| **WS1b** | пер-модельный rate-guard (семафор конкуренции + пейсинг) | ✅ **СДАНО** (concurrency-тест) |
|
||||||
|
| **WS1c-инфра** | base/enriched `MemoryVersion`-split (механика пер-волнового снапшота, «переоплата ОДНА») | ✅ **СДАНО** (unit-тест) |
|
||||||
|
| **WS4-core** | банкнота: pure `splitBanknote`/`parseBanknote` (`⟦TM-BANK-v1⟧`, толерантный сплит, Han-в-src, finish-truncation) | ✅ **СДАНО** (7 инвариант-тестов; Go↔Python паритет к `banknote.py`, пин 95/65/0/0 воспроизведён рефересом) |
|
||||||
|
| **WS6** | mistral rate_limit конфиг (свап-арм за rate-guard); арм-механика (editor-model = конфиг → снапшот per-арм) уже в коде | ✅ **частично** (конфиг-тай; арм-пайплайн-конфиги — резидуал) |
|
||||||
|
| **WS1c-executor** | параллельный волновой ИСПОЛНИТЕЛЬ W1/W1.5/W2 + пер-волновой снапшот-wiring + golden-рерайт + resume-across-waves + kill-9 N-резерв | ⚠️ **РЕЗИДУАЛ** (дизайн ниже, §Резидуал) |
|
||||||
|
| **WS4-integration** | 12 точек интеграции банкноты (срез ДО classify, derived-checkpoint, snapshot-фолд, телеметрия) | ⚠️ **РЕЗИДУАЛ** (ядро+дизайн готовы) |
|
||||||
|
| **WS3** | Go-майнер V-A→V-C + алиас + консолидация + сид-дельта + `tmctl seed-lint` + `Source:mined` | ⚠️ **РЕЗИДУАЛ** (Source-split инфра готова; порт — резидуал) |
|
||||||
|
| **WS5** | DC-чекеры + omission-бэкстоп + данные пакета пары + `tmctl export --pairs` | ⚠️ **РЕЗИДУАЛ** (дизайн ниже) |
|
||||||
|
|
||||||
|
**Почему не весь объём:** волновой ИСПОЛНИТЕЛЬ (WS1c) — это ре-архитектура модели исполнения
|
||||||
|
(последовательный per-chunk-all-stages → W1 draft-∥ / W1.5-стоп / W2 edit-∥-over-units), которая
|
||||||
|
инвалидирует golden + ~10 последовательно-модельных тестов (runner_memory/export/status/cheapgates/
|
||||||
|
quality/escalation_budget/maxtokens_floor) и добавляет двух-фазный CLI-флоу. Довести это до зелёного
|
||||||
|
за сессию БЕЗ поломки суита нереалистично. Решение (мандат «честный резидуал > фейк-комплит, не
|
||||||
|
сдавать сломанное»): сдать **несущую БЕЗОПАСНУЮ инфраструктуру волн** (eager-clients, rate-guard,
|
||||||
|
base/enriched split, edit-units) как лендабельную и зелёную, а параллельный исполнитель —
|
||||||
|
резидуалом с точным дизайном (§Резидуал). Ни один сданный кусок не оставляет суит красным.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. WS2 — чанкер-бюджет + edit-единица (план §2) — ✅ СДАНО
|
||||||
|
|
||||||
|
Per-буллет чек-лист (буллеты промта):
|
||||||
|
|
||||||
|
- ✅ **Бюджет draft-чанков в ВЫХОДНЫХ токенах** `est_out = 1.1978·cjk + 0.3852·other`, конфиг
|
||||||
|
`config.Segmentation` (per-pair, снапшот-folded через `segmentationSnap`). Классы символов —
|
||||||
|
**РЕАЛЬНЫЕ `unicode.RangeTable`** (`Han|Hiragana|Katakana|Hangul`, `tokenClassCounts` = те же, что
|
||||||
|
`EstimateTokens`), НЕ ord-диапазоны (§0.1 общность). Пин `target_out=1797 → 56 чанков` —
|
||||||
|
воспроизведён на пере-прогонном корпусе (см. §Верификация ниже).
|
||||||
|
- ✅ **Edit-единица = глава; дефолт `EditCeilingOut=3200`.** Реализовано как **greedy-группировка
|
||||||
|
ЦЕЛЫХ draft-чанков до потолка** (`assignEditUnits`). **ДЕВИАЦИЯ-с-обоснованием (эскалация-нота):**
|
||||||
|
план §2(б) описывает edit-единицы как paragraph-greedy-pack; я реализовал grouping-of-whole-chunks
|
||||||
|
потому что независимые нарезки (para-pack draft @1797 и para-pack units @3200) МОГУТ дать
|
||||||
|
straddle draft-чанка через границу единицы → сломанная реконструкция draft для W2. **Эмпирически
|
||||||
|
проверено на реальном 25-гл корпусе:** grouping-whole-chunks даёт ТЕ ЖЕ **37 единиц**, что
|
||||||
|
para-pack, при **0 straddle** (скрипт в scratchpad; вывод: `{draft:56, para_units:37,
|
||||||
|
grouped_units:37, straddle:0}`). Т.е. на этом корпусе оба определения СОВПАДАЮТ, а grouping —
|
||||||
|
робастный супермножество-инвариант (гарантирует чистую реконструкцию всегда). Пин 37 сохранён.
|
||||||
|
- ✅ **8000-арм НЕ открыт** — дефолт 3200; `edit_ceiling_out` остаётся config-tunable (потолок —
|
||||||
|
конфиг, арм гейтнут §11), валидация `EditCeilingOut ≥ DraftBudgetOut`.
|
||||||
|
- ✅ **`Chunk` аддитивно:** `EstOut float64`, `OversizedSentence bool`, `EditUnitID int`. Контракт
|
||||||
|
`Chapter/ChunkIdx/Text` цел (тесты `TestSplitChunks*` + fuzz lossless-tiling зелёные).
|
||||||
|
- ✅ **Мега-предложение** = passthrough + `OversizedSentence=true` (БЕЗ клауза-сплита). Пин
|
||||||
|
`TestSplitChunksOversizeSentenceIsOwnChunk`.
|
||||||
|
- ✅ **Снос мёртвых `STMDepth`/`OverlapTokens`** из `config.ContextAssembly` + `contextSnap` +
|
||||||
|
yaml-конфигов (carryover не строим). Структурная смена `contextSnap` → строка манифеста §8.
|
||||||
|
- ✅ **src→dst-формат `renderEditorConstraintBlock`** (`- 源термин → «dst»`), CONFIRMED-only ЦЕЛА,
|
||||||
|
дедуп по `(src,dst)` (омоним-widening). Обоснование — билингв-редактор + омонимичный dst (§2в
|
||||||
|
ревью-1 F4). **Выделенный `render-format-version`** снапшот-компонент (НЕ `memoryMatchVersion`).
|
||||||
|
Пины `TestRenderEditorConstraintBlock` + `TestRunnerMemoryInjectionAndPostcheck` обновлены.
|
||||||
|
- ✅ **Сайзинг max_tokens редактора** — НЕ трогал (F8: существующий `stagerun.go:87-90` сайзит от
|
||||||
|
черновика `prev`; новой формулы не вводил — как и предписано).
|
||||||
|
- ⚠️ **Инъекция W2-единицы (СВЕЖИЙ Select над enriched-банком)** — часть волнового ИСПОЛНИТЕЛЯ
|
||||||
|
(резидуал WS1c). Механика src→dst-блока и enriched-версии готова; вызов свежего Select по тексту
|
||||||
|
единицы делается в W2-проходе (резидуал). **Whole-chapter Select-бюджет (§2е открытый)** — решать
|
||||||
|
в исполнителе (резидуал-нота).
|
||||||
|
|
||||||
|
Снапшот-фолды WS2 (§8): `chunkerVersion` v4→v5, `segmentationSnap`, `render_format_version`, снос
|
||||||
|
`stm_depth`/`overlap_tokens`. Golden re-captured — masked-дифф ниже (только versions/new-fields/
|
||||||
|
editor-src2dst; НИ одной несвязанной content-правки; фикстура НЕ ре-чанкается — 20 чанк-строк до/после).
|
||||||
|
|
||||||
|
## 2. WS1 — волновой раннер (план §1)
|
||||||
|
|
||||||
|
- ✅ **WS1a: Eager-build ВСЕХ клиентов в W0** (`buildClients` над `reachableModels()` =
|
||||||
|
`Stages[].Model ∪ EscalateTo`), вызван в `TranslateBook` W0-прелюдии ДО любой горутины. `client()`
|
||||||
|
теперь **READ-ONLY** — громкий отказ на miss (закрывает гонку `runner.go:171-181` без лока;
|
||||||
|
трипваер на будущую 3-ю модель-ось). `BuildClient` не требует ключа → безопасно на key-less resume.
|
||||||
|
`-race` чист.
|
||||||
|
- ✅ **WS1b: Пер-модельный rate-guard** (`ratelimit.go`): семафор конкуренции `models.yaml
|
||||||
|
rate_limit.max_concurrency` + опц. пейсинг `min_interval_ms`. **Транспорт-ось, НЕ снапшот** (wire
|
||||||
|
не трогает). Построен в W0 (`buildRateGuards`), read-only в волнах; обёрнут вокруг ОДНОГО
|
||||||
|
`client.Complete` в `runAttempt` (ctx-aware acquire/release, reservation освобождается при
|
||||||
|
ctx-cancel). Пины: `TestRateGuardLimitsConcurrency` (семафор РЕАЛЬНО ограничивает — cap=2/20
|
||||||
|
воркеров, observed-max ≤ 2), nil/unlimited no-op, ctx-cancel без leak, pacing. **`waves.workers`
|
||||||
|
конфиг** — вводится с исполнителем (резидуал).
|
||||||
|
- ✅ **WS1c-инфра: base/enriched `MemoryVersion`-split.** `MemoryBank.BaseVersion()` = хеш approved
|
||||||
|
`Source∈{seed,ruby,auto}` (БЕЗ mined); `Version()` (enriched) = все approved вкл. `Source:mined`.
|
||||||
|
`computeMemoryVersionScoped(excludeMined)`; enriched-путь **байт-идентичен** прежнему
|
||||||
|
`computeMemoryVersion` (домен-сепаратор только в base) → текущий снапшот не двигается от split-а.
|
||||||
|
Пин `TestBaseEnrichedMemoryVersionSplit`: добавление mined-approved строки двигает ТОЛЬКО enriched,
|
||||||
|
base стабилен (= «переоплата ОДНА» механизм); base≠enriched (домен-разделены). **`Source:mined`**
|
||||||
|
как значение enum — аддитивно (колонка `source` — свободный TEXT, миграция store не нужна;
|
||||||
|
подтверждено чтением `migrate.go`).
|
||||||
|
- ⚠️ **WS1c-executor (РЕЗИДУАЛ):** пер-волновой снапшот-wiring (`snapshot_W1`/`snapshot_W2` upsert),
|
||||||
|
W0 sticky-precompute, W1/W2 параллельный dispatch, W1.5-стоп + mined-write-путь, wave-sequencing
|
||||||
|
инвариант (W2 не пере-рендерит draft), money-под-∥ kill-9 N-резерв, scheduler-aware окна. Дизайн —
|
||||||
|
§Резидуал.
|
||||||
|
|
||||||
|
## 3. WS4-core — банкнота-парсер (план §4б) — ✅ СДАНО
|
||||||
|
|
||||||
|
`internal/pipeline/banknote.go` — байт-верный порт `exp16/banknote.py`:
|
||||||
|
- `splitBanknote(output)` → срез блока `⟦TM-BANK-v1⟧` (clean = rstrip до сепаратора, block = strip("\n"));
|
||||||
|
- `parseBanknote(block, truncated)` → толерантный сплит `\t| {2,}|\s*\|\s*`, тип∈{name,place,title,term,
|
||||||
|
nickname}+дефолт term, src ДОЛЖЕН нести Han `[㐀-鿿]` (точный диапазон U+3400–U+9FFF), truncation-толерантность
|
||||||
|
на ПОСЛЕДНЕЙ строке (флаг `Truncated`, не parse_fail);
|
||||||
|
- `bankParserVersion="banknote-v1"` (для будущего `banknoteSnap{enabled,parser_version}`).
|
||||||
|
- Тесты (`banknote_test.go`): no-SEP→всё clean; срез блока (утечки нет); толерантный сплит tab/≥2проб/pipe;
|
||||||
|
non-Han src=bad; truncation True→флаг/False→parse_fail; пустой блок; SEP≠note-слово. **Все PASS.**
|
||||||
|
- **Паритет:** референс `ws4_banknote_verify.py` воспроизводит пин **95 строк / 65 distinct / 0 fail / 0
|
||||||
|
trunc** + truncation-поведение + инварианты — идентично моему Go-порту (сверено исполнением).
|
||||||
|
|
||||||
|
**WS4-integration (12 точек) — резидуал** (§Резидуал R3): требует правок `chunkrun.go`/`stagerun.go`/
|
||||||
|
`disposition.go`/`snapshot.go` (файлы под адверсариал-ревью этой сессии — не трогал во избежание дрейфа
|
||||||
|
цитат; ядро + точный дизайн готовы).
|
||||||
|
|
||||||
|
## 4–5. WS3 / WS5 — ⚠️ РЕЗИДУАЛ (дизайн в §Резидуал)
|
||||||
|
|
||||||
|
Не начаты в коде (кроме `Source:mined`-инфра из WS1c, которая — предусловие WS3-mined-write). Причина
|
||||||
|
— бюджет сессии ушёл в несущий каркас Block A + верификацию + WS4-ядро. Дизайны/пред-условия — §Резидуал.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Багфиксы вне плана
|
||||||
|
Пока нет (изменения строго по плану). Правки по ревью — см. §Адверсариал-ревью ниже (2 doc-фикса + 1 gofmt).
|
||||||
|
|
||||||
|
## Структурные переносы
|
||||||
|
- Новый файл `internal/pipeline/ratelimit.go` (rate-guard) + `banknote.go` (WS4-ядро) — новые оси, не рефактор.
|
||||||
|
- `SegBudget` (resolved segmentation) живёт в `chunker.go`; резолвер `segBudget()` — в `runner.go`
|
||||||
|
(segmenter остаётся config-free). `renderFormatVersion` const — в `memory.go` рядом с рендерерами.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Верификация (исполнением — логи, не слова)
|
||||||
|
|
||||||
|
```
|
||||||
|
$ go build ./... → OK
|
||||||
|
$ go vet ./... → OK (чисто)
|
||||||
|
$ go test -race ./... → ВСЁ ЗЕЛЁНОЕ:
|
||||||
|
ok cmd/tmctl 1.0s · config · ledger · llm · pipeline 19.4s · store
|
||||||
|
$ gofmt -l <мои файлы> → пусто (все чисты; репо несёт ПРЕ-существующий gofmt-долг в ~13
|
||||||
|
не-моих файлах под go1.26.4 — их не трогал)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Фикстуры плана (исполнены рефересами `eval/design11/*` под `eval/.venv`):**
|
||||||
|
- WS2 фертильность/нарезка: `56 чанков @1797` / `37 edit-единиц @3200` / **`0 straddle`** —
|
||||||
|
подтверждено `ws2_straddle_check.py` (grouping-whole-chunks ≡ para-pack на корпусе).
|
||||||
|
- WS4 банкнота: `95 строк / 65 distinct / 0 fail / 0 trunc` (`ws4_banknote_verify.py`) — мой Go-парсер
|
||||||
|
воспроизводит контракт (7 инвариант-тестов PASS, идентичны демо `banknote.py`).
|
||||||
|
|
||||||
|
**Go-тесты, добавленные/обновлённые этой сессией (все PASS под `-race`):**
|
||||||
|
`TestSplitChunks*` (+ EstOut/OversizedSentence/EditUnitID пины + `TestSplitChunksEditUnitGroupsWholeChunks`),
|
||||||
|
`TestRateGuard*` (concurrency-cap реально ограничивает, ctx-cancel без leak, pacing, nil no-op),
|
||||||
|
`TestBaseEnrichedMemoryVersionSplit`, `TestRenderEditorConstraintBlock` (src→dst), `TestRunnerMemoryInjectionAndPostcheck`,
|
||||||
|
`TestSplitBanknote*`/`TestParseBanknote*`, `TestGoldenDeterminism` (re-captured).
|
||||||
|
|
||||||
|
## Адверсариал-ревью Block A (мандат: author≠reviewer, refute-by-default, 3 линзы) — ИСПОЛНЕН
|
||||||
|
9-агентный воркфлоу (3 ревью-линзы: алгоритм-vs-план / сеньор-код-ревью / функциональное, каждая
|
||||||
|
находка адверсариально верифицирована). **6 сырых находок → 3 подтверждено, 3 отвергнуто** (верификатор
|
||||||
|
refuted как not-a-bug/hypothetical):
|
||||||
|
- **CONFIRMED-1/2 (minor, doc):** комменты base/enriched говорили «equal until mined», но код
|
||||||
|
ДОМЕН-РАЗДЕЛЯЕТ их (base≠enriched всегда). Код/дизайн ВЕРНЫ; комменты — нет. **ПОФИКШЕНО**
|
||||||
|
(`memory.go:135`,`:179`).
|
||||||
|
- **CONFIRMED-3 (nit, style):** `subChunk` struct не gofmt-выровнен. **ПОФИКШЕНО** (`gofmt -w`).
|
||||||
|
- **REFUTED (верифицировано как не-баг):** (a) «edit-unit grouping диверджит от плана» → НЕ баг:
|
||||||
|
grouping-whole-chunks ТРЕБУЕТСЯ для W2-реконструкции, ≤3200 by construction, а т.к.
|
||||||
|
`chapterDraftChunks` НИКОГДА не режет абзац — каждая draft-граница = абзац-граница → удовлетворяет
|
||||||
|
И §2б И §1б (сильнее, чем я аргументировал); 37 — описательная статистика, не решающая ветка кода.
|
||||||
|
(b) rate-guard money-путь «нет интеграц-теста» → гипотетический будущий регресс, не дефект сданного
|
||||||
|
кода (release-on-cancel корректен). (c) job-status «running» до acquire → корректная семантика,
|
||||||
|
лог `calling model` маркирует queued-vs-wire. Диспозиции — в отчёте, тихих девиаций нет.
|
||||||
|
|
||||||
|
## Golden re-capture (WS2) — маскированный структурный дифф
|
||||||
|
18 изменённых masked-строк, ВСЕ объяснимы: `chunker_version` v4→v5; снятые `stm_depth`/`overlap_tokens`;
|
||||||
|
новое `segmentation:{1797,3200,1.1978,0.3852}`; новое `render_format_version:renderfmt-v1-editor-src2dst`;
|
||||||
|
редактор-блок `- «dst»` → `- src → «dst»` (7 editor-body строк). Фикстура НЕ ре-чанкается. Полный
|
||||||
|
masked-дифф — scratchpad `ws2_golden_masked_diff.txt`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## РЕЗИДУАЛ — точные дизайны для следующей сессии
|
||||||
|
|
||||||
|
### R1. Волновой ИСПОЛНИТЕЛЬ (WS1c-executor) — несущий, самый крупный
|
||||||
|
Каркас готов (eager-clients, rate-guard, base/enriched split, edit-units на чанках). Осталось:
|
||||||
|
1. `snapshotIDForWave(wave)` — рефактор `snapshot.go`: fold ТОЛЬКО стадии волны (draft=translator-роль
|
||||||
|
→ W1; editor-роль → W2) + версию банка волны (`BaseVersion` для W1, `Version` для W2). Апсертить
|
||||||
|
ДВА снапшота.
|
||||||
|
2. Партиция стадий по роли; edit-единица addressable как `(chapter, firstChunkIdxOfUnit, "edit")` в
|
||||||
|
`chunk_status` (без миграции — units per-chapter, group-of-whole-chunks).
|
||||||
|
3. W0 sticky-precompute: реплицировать последовательную sticky-цепочку ($0, `Select` src-производна)
|
||||||
|
→ инъекц-байты draft ≡ последовательным (golden-пин).
|
||||||
|
4. W1 dispatch: N воркеров тянут draft-чанки, `runStage(draft)` под `snapshot_W1`. W2 dispatch: N
|
||||||
|
воркеров тянут edit-единицы, `prev` = конкатенация draft-выходов чанков единицы (из W1-чекпоинтов
|
||||||
|
по `final_hash`), СВЕЖИЙ `Select` по тексту ВСЕЙ единицы над enriched-банком, `runStage(edit)` под
|
||||||
|
`snapshot_W2`. **W2 НИКОГДА не пере-рендерит draft** (несущий инвариант §1в).
|
||||||
|
5. W1.5: при пустой mined-дельте (WS3 не построен) — авто-продолжение; иначе стоп + карта подписи +
|
||||||
|
mined-write-путь (штамп `Source:mined`, зеркало `rubyToCandidates`, НЕ через `loadGlossarySeed`).
|
||||||
|
6. Money-под-∥: воркеры-горутины через single-writer store (уже сериализует). kill-9 тест с N живыми
|
||||||
|
резервами (порт `store/kill9_test.go` на N). rate-guard уже интегрирован.
|
||||||
|
7. Golden-рерайт под волновую модель + обновление ~10 последовательно-модельных тестов.
|
||||||
|
|
||||||
|
### R2. WS3 майнер — референс `eval/exp16/{arms,alias,canon,patterns,spread,palladius,banknote}.py`
|
||||||
|
Новый пакет `internal/pipeline/miner*.go`. Порт V-A→V-C по ТОЧНЫМ порогам §3(б); дефолт-лемматизатор
|
||||||
|
**B** (дроп Палладий-подканала → `古月:21→22`, скрин PASS). Инвариантные гарантии (EXACT): 13618-канд-СЕТ,
|
||||||
|
recall 0.9649/0.9318, катастроф-скрин `方源/蛊/蛊师` ранги 0/1/2 ∈top-50. Реюз `memnorm.normalizeSourceKey`.
|
||||||
|
`tmctl seed-lint` (сухой прогон фейл-лаудов `loadGlossarySeed`). Go↔Python паритет (Python=референс).
|
||||||
|
`Source:mined`-инфра (base/enriched split) — **готова** (WS1c).
|
||||||
|
|
||||||
|
### R3. WS4 банкнота — референс `eval/exp16/banknote.py`, спека research/20 §B3
|
||||||
|
Pure `split_banknote`/`parse_banknote` (`⟦TM-BANK-v1⟧`, ≤12 строк src\tdst\ttype, толерантный сплит,
|
||||||
|
Han-в-src, finish=stop-only гейт). 12 точек: срез ДО `classifyOutput` (`chunkrun.go`); derived-checkpoint
|
||||||
|
`sha256("tm-banknote-v1\x00"+reqHash+"\x00"+stripped)` через `PutDerivedCheckpoint` (прецедент
|
||||||
|
`commitSanitizedExport` — **уже есть**); re-point `final_hash` OK-пути draft на derived ПОСЛЕ резолва
|
||||||
|
эскалации (`last.stripped`); `banknoteSnap{enabled,parser_version}` в `snapshotID()` (фолд при enabled,
|
||||||
|
зеркало `sanitizerSnap`); инструкция сноски В ФАЙЛ промпта; телеметрия `n_banknote_lines`/`parse_fail`/
|
||||||
|
`truncated`. Пин: пере-парс 95 строк = 0 fail (`ws4_banknote_verify.py`).
|
||||||
|
|
||||||
|
### R4. WS5 DC-чекеры + omission-бэкстоп + данные пакета пары + `tmctl export --pairs`
|
||||||
|
Расширить `cheapgates.go` DC1/2/6/7 (детерм. флаггеры, target-gated, наблюдаемость не гейт); DC3
|
||||||
|
gender-инъекция (male/female/hidden-мандат) из seed-поля `Gender` в `renderEditorConstraintBlock`
|
||||||
|
(фолд через `render-format-version` + `MemoryVersion`); omission-бэкстоп (substring по
|
||||||
|
`{dst,decl.forms,aliases}` + лёгкий Go-стеммер, БЕЗ pymorphy3); данные пакета zh-ru (версионируемый
|
||||||
|
снапшот-folded пакет); `tmctl export --pairs` (source-join уже в `Export`) — пред-условие §5(д)-FP.
|
||||||
|
DC5-детекцию НЕ строить (74%/42% FP); hidden-ЧЕКЕР НЕ строить (Ф2-кореф). Референс `ws5_checkers_verify.py`.
|
||||||
|
|
||||||
|
### R5. WS6 арм-конфиги
|
||||||
|
Арм-механика (editor-model = конфиг → `stageSnap.Model` → снапшот per-арм) **уже работает**. Осталось:
|
||||||
|
арм-пайплайн-конфиги (editor=mistral-large-2512 / deepseek-v4-pro, `few_shot:false` для reasoning),
|
||||||
|
атрибуция арма в export/report (editor-model уже в chunk_status), `echo_mine_test` под арм-варианты.
|
||||||
|
mistral rate_limit — **сдан** (WS1b/WS6 тай).
|
||||||
|
|
||||||
|
### Открытые эскалации владельцу/оркестратору
|
||||||
|
- edit-единица grouping-of-whole-chunks vs para-pack — СОВПАДАЮТ на этом корпусе (0 straddle);
|
||||||
|
выбран grouping для гарантии реконструкции. Требует подтверждения при лендинге.
|
||||||
|
- Реестр §10 (W1.5-UX, DeepSeek ×2 ре-чек 24.07, whole-chapter Select-бюджет) — открыты как в плане.
|
||||||
|
|
@ -218,10 +218,16 @@ models:
|
||||||
# = $0.5/$1.5 (OpenRouter согласен) → разрешает D19-подозрение ($0.5/$1.5 при $2/$6 = 4×). cached=input
|
# = $0.5/$1.5 (OpenRouter согласен) → разрешает D19-подозрение ($0.5/$1.5 при $2/$6 = 4×). cached=input
|
||||||
# консервативно (cache-read не моделируем; канал B — свежий вход/чанк, hit≈0, потолок не слепнет).
|
# консервативно (cache-read не моделируем; канал B — свежий вход/чанк, hit≈0, потолок не слепнет).
|
||||||
price: { input_per_m: 0.5, cached_per_m: 0.5, cache_write_per_m: 0, output_per_m: 1.5 }
|
price: { input_per_m: 0.5, cached_per_m: 0.5, cache_write_per_m: 0, output_per_m: 1.5 }
|
||||||
|
# WS1 §1б / WS6 gate №4: mistral-large — агрессивный rate-limiter (~48% retry-fails под N-∥ @1.3s;
|
||||||
|
# токен-бакет/конкуренц-кап, тир-зависим — 00-provider-quirks §Транспорт). Пер-модельный семафор
|
||||||
|
# конкуренции волнового раннера ограничивает одновременные вызовы (транспорт-ось, НЕ снапшот).
|
||||||
|
# Значение — консервативный старт; пере-замерить на прод-тире (правило двух направлений). Свап-арм
|
||||||
|
# редактора mistral НЕ идёт через прод-путь без этого guard (ревью-2 F4).
|
||||||
|
rate_limit: { max_concurrency: 2, min_interval_ms: 0 }
|
||||||
note: >-
|
note: >-
|
||||||
Переводчик-дефолт канала B (D19.1): 10/10 чисто на violence, policy 11.06.2026 без запрета adult, не
|
Переводчик-дефолт канала B (D19.1): 10/10 чисто на violence, policy 11.06.2026 без запрета adult, не
|
||||||
reasoning-модель (эхо-мины нет). Полный wiring канала B — Ф2 (конфиг-слот). Полигон перепроверяет цену (D19.5б).
|
reasoning-модель (эхо-мины нет). Свап-арм редактора (WS6, за rate-guard). Полный wiring канала B — Ф2.
|
||||||
Живьём 2026-07-10: /models ✓, chat 200.
|
Полигон перепроверяет цену (D19.5б). Живьём 2026-07-10: /models ✓, chat 200.
|
||||||
|
|
||||||
gpt-5-mini: # кандидат-редактор/second-opinion судья (D3) — НЕ в дефолтной цепочке Ф1
|
gpt-5-mini: # кандидат-редактор/second-opinion судья (D3) — НЕ в дефолтной цепочке Ф1
|
||||||
provider: openai
|
provider: openai
|
||||||
|
|
|
||||||
|
|
@ -22,9 +22,20 @@ context:
|
||||||
# DeepSeek $0.0028/M — решается расчётом на прототипе).
|
# DeepSeek $0.0028/M — решается расчётом на прототипе).
|
||||||
glossary_injection: selective
|
glossary_injection: selective
|
||||||
glossary_token_budget: 800
|
glossary_token_budget: 800
|
||||||
stm_depth: 2 # глубина STM в чанках (Фаза 1)
|
|
||||||
overlap_tokens: 200 # перекрытие чанков = read-only контекст (Фаза 1)
|
|
||||||
cache_ttl: "5m"
|
cache_ttl: "5m"
|
||||||
|
# STMDepth/OverlapTokens сняты (WS2 §2а): carryover/overlap НЕ строится (D39.7/8 — метрики
|
||||||
|
# под полом когезии 0.126); мёртвые кнобы в снапшоте приглашали тихий «re-activate».
|
||||||
|
|
||||||
|
# WS2 (слой 1): бюджет нарезки в ВЫХОДНЫХ (ru) токенах через фертильность (est_out = cjk·CJK +
|
||||||
|
# other·Other); снапшот-folded (segmentationSnap → громкий --resnapshot при правке). Пропуск блока =
|
||||||
|
# ратифицированные zh-ru дефолты (см. LoadPipeline). Крупно-главный арм (edit_ceiling_out>3200)
|
||||||
|
# гейтнут платными Q2a span-судьями (§11) — 3200 = консервативный дефолт.
|
||||||
|
segmentation:
|
||||||
|
draft_budget_out: 1797 # мелкий DRAFT-чанк, ru-выход (→ 56 чанков на 25-гл пере-прогоне)
|
||||||
|
edit_ceiling_out: 3200 # крупная EDIT-единица = глава/группа целых чанков (→ 37 единиц)
|
||||||
|
fertility: # независимо пере-выведено на пере-прогоне: R²=0.9633
|
||||||
|
cjk: 1.1978
|
||||||
|
other: 0.3852
|
||||||
|
|
||||||
retries:
|
retries:
|
||||||
# Содержательные регенерации после провала гейта ДО эскалации (транспортные
|
# Содержательные регенерации после провала гейта ДО эскалации (транспортные
|
||||||
|
|
|
||||||
|
|
@ -12,9 +12,8 @@ defaults:
|
||||||
context:
|
context:
|
||||||
glossary_injection: selective
|
glossary_injection: selective
|
||||||
glossary_token_budget: 800
|
glossary_token_budget: 800
|
||||||
stm_depth: 2
|
|
||||||
overlap_tokens: 200
|
|
||||||
cache_ttl: "5m"
|
cache_ttl: "5m"
|
||||||
|
# STMDepth/OverlapTokens сняты (WS2 §2а — carryover не строим).
|
||||||
|
|
||||||
retries:
|
retries:
|
||||||
regenerate_before_escalate: 1
|
regenerate_before_escalate: 1
|
||||||
|
|
|
||||||
|
|
@ -130,6 +130,19 @@ type Model struct {
|
||||||
// Capabilities overrides this model's provider-default wire shape (D3.1),
|
// Capabilities overrides this model's provider-default wire shape (D3.1),
|
||||||
// field-by-field (model wins). Nil = inherit the provider/kind baseline.
|
// field-by-field (model wins). Nil = inherit the provider/kind baseline.
|
||||||
Capabilities *CapabilitiesConfig `yaml:"capabilities"`
|
Capabilities *CapabilitiesConfig `yaml:"capabilities"`
|
||||||
|
// RateLimit caps how many parallel wave workers may call THIS model at once (WS1 §1б, ревью-2
|
||||||
|
// F4): mistral-large-latest fails ~48% of calls under N-parallelism (a token-bucket / concurrency
|
||||||
|
// cap, tier-dependent — quirks §Транспорт) while grok is 0%, so a model can bound its own
|
||||||
|
// wave concurrency. Optional min_interval paces call STARTS. A TRANSPORT axis — wire-neutral,
|
||||||
|
// NOT snapshot-folded (it never touches the request bytes). Zero = unlimited (the default).
|
||||||
|
RateLimit RateLimit `yaml:"rate_limit"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RateLimit is the per-model wave-concurrency guard (WS1). max_concurrency 0 = unlimited;
|
||||||
|
// min_interval_ms 0 = no pacing.
|
||||||
|
type RateLimit struct {
|
||||||
|
MaxConcurrency int `yaml:"max_concurrency"`
|
||||||
|
MinIntervalMS int `yaml:"min_interval_ms"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Price mirrors ledger.ModelPrice in YAML form (USD per 1M tokens).
|
// Price mirrors ledger.ModelPrice in YAML form (USD per 1M tokens).
|
||||||
|
|
@ -210,6 +223,9 @@ func LoadModels(path string) (*Models, error) {
|
||||||
bad("model %s: non-local models require non-zero input/output prices", name)
|
bad("model %s: non-local models require non-zero input/output prices", name)
|
||||||
}
|
}
|
||||||
validateCapabilities(bad, "model "+name, mod.Capabilities)
|
validateCapabilities(bad, "model "+name, mod.Capabilities)
|
||||||
|
if mod.RateLimit.MaxConcurrency < 0 || mod.RateLimit.MinIntervalMS < 0 {
|
||||||
|
bad("model %s: rate_limit.max_concurrency/min_interval_ms must be ≥0 (a concurrency cap / pacing interval)", name)
|
||||||
|
}
|
||||||
if why := m.echoMineViolation(name); why != "" {
|
if why := m.echoMineViolation(name); why != "" {
|
||||||
bad("model %s on echo-prone provider %s: %s — this re-arms the DeepSeek echo mine "+
|
bad("model %s on echo-prone provider %s: %s — this re-arms the DeepSeek echo mine "+
|
||||||
"(the provider echoes the untranslated CJK source when thinking is OFF; PROGRESS «Ответ Полигону», §2); "+
|
"(the provider echoes the untranslated CJK source when thinking is OFF; PROGRESS «Ответ Полигону», §2); "+
|
||||||
|
|
|
||||||
|
|
@ -19,15 +19,43 @@ import (
|
||||||
|
|
||||||
// Pipeline is the parsed pipeline-*.yaml.
|
// Pipeline is the parsed pipeline-*.yaml.
|
||||||
type Pipeline struct {
|
type Pipeline struct {
|
||||||
Core string `yaml:"core"` // C0|C1|C2|C3
|
Core string `yaml:"core"` // C0|C1|C2|C3
|
||||||
Version int `yaml:"version"`
|
Version int `yaml:"version"`
|
||||||
Defaults PipelineDefaults `yaml:"defaults"`
|
Defaults PipelineDefaults `yaml:"defaults"`
|
||||||
Context ContextAssembly `yaml:"context"`
|
Context ContextAssembly `yaml:"context"`
|
||||||
Retries Retries `yaml:"retries"`
|
Segmentation Segmentation `yaml:"segmentation"`
|
||||||
Stages []Stage `yaml:"stages"`
|
Retries Retries `yaml:"retries"`
|
||||||
Gates Gates `yaml:"gates"`
|
Stages []Stage `yaml:"stages"`
|
||||||
Escal Escalation `yaml:"escalation"`
|
Gates Gates `yaml:"gates"`
|
||||||
Fanout Fanout `yaml:"fanout"`
|
Escal Escalation `yaml:"escalation"`
|
||||||
|
Fanout Fanout `yaml:"fanout"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Segmentation is the WS2 OUTPUT-token chunking budget (слой 1, L2-budget-wrong-unit fix): the
|
||||||
|
// draft-chunk and edit-unit ceilings in ru-OUTPUT tokens, plus the per-pair fertility coefficients
|
||||||
|
// that convert source char-classes to an output-token estimate (est_out = cjk·CJK + other·Other).
|
||||||
|
// Snapshot-folded (segmentationSnap), so a budget/fertility edit is a loud --resnapshot (D30.9).
|
||||||
|
// The char-class CLASSIFIER is the backend's real unicode ranges (EstimateTokens: Han|Hiragana|
|
||||||
|
// Katakana|Hangul) for generality (§0.1) — the coefficients are the only per-pair datum (authored
|
||||||
|
// per project). Zero fields fall to the ratified zh-ru defaults in LoadPipeline.
|
||||||
|
type Segmentation struct {
|
||||||
|
// DraftBudgetOut is the fine DRAFT-chunk ceiling (ru-output tokens, default 1797 → 56 chunks on
|
||||||
|
// the 25-chapter rerun). A draft chunk is the small unit for COGS/coverage/alignment.
|
||||||
|
DraftBudgetOut int `yaml:"draft_budget_out"`
|
||||||
|
// EditCeilingOut is the coarse EDIT-unit ceiling (ru-output tokens, default 3200 → 37 units).
|
||||||
|
// The edit unit is a chapter (or a greedy grouping of whole draft chunks when a chapter exceeds
|
||||||
|
// the ceiling) — the large unit the reflow editor needs for cross-chunk cohesion (D39 п.4). The
|
||||||
|
// large-chapter arm (>3200, up to 8000) is GATED behind paid Q2a span-judges (§11); the ceiling
|
||||||
|
// stays config-tunable but 3200 is the conservative ratified default (span-omission unproven).
|
||||||
|
EditCeilingOut int `yaml:"edit_ceiling_out"`
|
||||||
|
Fertility Fertility `yaml:"fertility"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fertility holds the output-token-per-source-char coefficients (WS2). Independently re-derived on
|
||||||
|
// the rerun corpus: cjk=1.1978, other=0.3852 (R²=0.9633). A recompute is a loud --resnapshot.
|
||||||
|
type Fertility struct {
|
||||||
|
CJK float64 `yaml:"cjk"`
|
||||||
|
Other float64 `yaml:"other"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// PipelineDefaults are cross-stage knobs.
|
// PipelineDefaults are cross-stage knobs.
|
||||||
|
|
@ -39,13 +67,13 @@ type PipelineDefaults struct {
|
||||||
}
|
}
|
||||||
|
|
||||||
// ContextAssembly holds the prompt-layout budgets (Р5-раскладка: стабильный
|
// ContextAssembly holds the prompt-layout budgets (Р5-раскладка: стабильный
|
||||||
// префикс / волатильный хвост — сборка Фазы 1; поля фиксируются сейчас, чтобы
|
// префикс / волатильный хвост). WS2 (§2а) removed the dead STMDepth/OverlapTokens knobs:
|
||||||
// схема конфига не менялась под ногами «Редакции»).
|
// carryover/overlap is NOT built (D39.7/8 — metrics under the 0.126 floor), and leaving
|
||||||
|
// no-op knobs in the wire snapshot invited a silent "re-activate" — so они снесены (a
|
||||||
|
// contextSnap structural change → §8 resnapshot manifest line).
|
||||||
type ContextAssembly struct {
|
type ContextAssembly struct {
|
||||||
GlossaryInjection string `yaml:"glossary_injection"` // selective | full_prefix (Р5: обе схемы)
|
GlossaryInjection string `yaml:"glossary_injection"` // selective | full_prefix (Р5: обе схемы)
|
||||||
GlossaryTokenBudget int `yaml:"glossary_token_budget"`
|
GlossaryTokenBudget int `yaml:"glossary_token_budget"`
|
||||||
STMDepth int `yaml:"stm_depth"`
|
|
||||||
OverlapTokens int `yaml:"overlap_tokens"`
|
|
||||||
CacheTTL string `yaml:"cache_ttl"` // per-stage TTL override — Фаза 1
|
CacheTTL string `yaml:"cache_ttl"` // per-stage TTL override — Фаза 1
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -285,6 +313,27 @@ func LoadPipeline(path string, models *Models) (*Pipeline, error) {
|
||||||
if p.Fanout.Candidates <= 0 {
|
if p.Fanout.Candidates <= 0 {
|
||||||
p.Fanout.Candidates = 1
|
p.Fanout.Candidates = 1
|
||||||
}
|
}
|
||||||
|
// WS2 segmentation defaults (ratified zh-ru, §2в): a config that omits the block gets the
|
||||||
|
// output-token budget + independently-re-derived fertility. Zero/negative → default.
|
||||||
|
if p.Segmentation.DraftBudgetOut <= 0 {
|
||||||
|
p.Segmentation.DraftBudgetOut = 1797
|
||||||
|
}
|
||||||
|
if p.Segmentation.EditCeilingOut <= 0 {
|
||||||
|
p.Segmentation.EditCeilingOut = 3200
|
||||||
|
}
|
||||||
|
if p.Segmentation.Fertility.CJK <= 0 {
|
||||||
|
p.Segmentation.Fertility.CJK = 1.1978
|
||||||
|
}
|
||||||
|
if p.Segmentation.Fertility.Other <= 0 {
|
||||||
|
p.Segmentation.Fertility.Other = 0.3852
|
||||||
|
}
|
||||||
|
// An edit unit is a grouping of WHOLE draft chunks, so the edit ceiling must be ≥ the draft
|
||||||
|
// budget — otherwise a single draft chunk already exceeds the unit ceiling and every unit is
|
||||||
|
// one chunk (the decoupling collapses). Loud config error, not a silent degenerate segmentation.
|
||||||
|
if p.Segmentation.EditCeilingOut < p.Segmentation.DraftBudgetOut {
|
||||||
|
bad("segmentation.edit_ceiling_out (%d) must be ≥ draft_budget_out (%d) — an edit unit groups whole draft chunks (WS2)",
|
||||||
|
p.Segmentation.EditCeilingOut, p.Segmentation.DraftBudgetOut)
|
||||||
|
}
|
||||||
switch p.Context.GlossaryInjection {
|
switch p.Context.GlossaryInjection {
|
||||||
case "", "selective", "full_prefix":
|
case "", "selective", "full_prefix":
|
||||||
default:
|
default:
|
||||||
|
|
|
||||||
135
backend/internal/pipeline/banknote.go
Normal file
135
backend/internal/pipeline/banknote.go
Normal file
|
|
@ -0,0 +1,135 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"unicode"
|
||||||
|
)
|
||||||
|
|
||||||
|
// banknote.go: the banknote-v1 in-band footnote channel (WS4, план §4 / research/20 §B3 / exp16
|
||||||
|
// banknote.py — RATIFIED D39.10 as the dst-delivery channel). The translator, AFTER the translation,
|
||||||
|
// MAY emit a versioned separator + tab-delimited term lines for NEW terms only (not in the injected
|
||||||
|
// glossary). This file is the pure, deterministic PARSER + SLICER — the wire-invariant core; the
|
||||||
|
// integration seams (slice BEFORE classifyOutput, derived checkpoint, snapshot fold, telemetry —
|
||||||
|
// план §4(а) 12 points) consume these functions. A faithful Go↔Python port (banknote.py is the
|
||||||
|
// reference): same separator, same tolerant field split, same Han-in-src rule, same truncation
|
||||||
|
// tolerance. Pin (ws4_banknote_verify.py): re-parse of 95 saved lines = 0 parse_fail / 0 truncated.
|
||||||
|
|
||||||
|
// bankSeparator marks the start of the banknote block. Chosen to (a) not occur in natural prose and
|
||||||
|
// (b) NOT be a "Примечание/Сноска/Комментарий" trailing-note word the sanitizer reserves (§B3-1), so
|
||||||
|
// the two channels never collide. Changing it is a wire/parser change → a loud --resnapshot (§4в).
|
||||||
|
const bankSeparator = "⟦TM-BANK-v1⟧"
|
||||||
|
|
||||||
|
// bankMaxLines is the per-chunk banknote line budget (§B3-5, feeds the max_tokens sizing at
|
||||||
|
// integration point 5). The parser does not enforce it (a model over-emitting is tolerated and
|
||||||
|
// merely over-counted); it bounds the prompt-side budget.
|
||||||
|
const bankMaxLines = 12
|
||||||
|
|
||||||
|
// bankParserVersion versions the split/parse algorithm — the verdict-axis component folded into
|
||||||
|
// banknoteSnap{enabled, parser_version} (§4в/§4а point 6), mirroring sanitizerSnap: a parser change
|
||||||
|
// re-resolves the stripped draft, so it must be a loud --resnapshot even without a prompt edit.
|
||||||
|
const bankParserVersion = "banknote-v1"
|
||||||
|
|
||||||
|
// bankTypeOK is the accepted type set; anything else falls back to "term" (a benign default).
|
||||||
|
var bankTypeOK = map[string]bool{"name": true, "place": true, "title": true, "term": true, "nickname": true}
|
||||||
|
|
||||||
|
// bankFieldSplit tolerates a tab, a run of ≥2 spaces, or a pipe (optionally whitespace-padded) as the
|
||||||
|
// field delimiter — the exact tolerance of banknote.py so a model that emits spaces instead of a real
|
||||||
|
// TAB still parses. (Go \s is ASCII-whitespace; banknote lines are tab/space-delimited, so the parity
|
||||||
|
// with Python's unicode \s around the pipe is exact on the term corpus.)
|
||||||
|
var bankFieldSplit = regexp.MustCompile(`\t| {2,}|\s*\|\s*`)
|
||||||
|
|
||||||
|
// bankEntry is one parsed candidate line (evidence for W1.5 §C; NOT written to the store until owner
|
||||||
|
// signoff). Dst is the model's proposed translation — the direct dst delivery the co-occurrence
|
||||||
|
// miner could not extract (蛊→гу, non-seed 龙公→Лун Гун).
|
||||||
|
type bankEntry struct {
|
||||||
|
Src string
|
||||||
|
Dst string
|
||||||
|
Type string
|
||||||
|
}
|
||||||
|
|
||||||
|
// bankFlags is the per-chunk telemetry surfaced loud at integration point 10.
|
||||||
|
type bankFlags struct {
|
||||||
|
NLines int // accepted (well-formed) banknote lines
|
||||||
|
ParseFail bool // any malformed residual line (non-Han src / <2 fields) that was NOT a tolerated truncation
|
||||||
|
Truncated bool // the LAST line was cut by generation length (tolerated, not a parse fail)
|
||||||
|
}
|
||||||
|
|
||||||
|
// splitBanknote slices the banknote block off the raw model output BEFORE any gate/editor (integration
|
||||||
|
// point 1). Returns (clean_translation, raw_block). No separator → the whole (right-trimmed) output is
|
||||||
|
// the clean translation and the block is "". Faithful to banknote.py: clean = output[:idx].rstrip(),
|
||||||
|
// block = output[idx+len(SEP):].strip("\n").
|
||||||
|
func splitBanknote(output string) (clean, block string) {
|
||||||
|
idx := strings.Index(output, bankSeparator)
|
||||||
|
if idx < 0 {
|
||||||
|
return strings.TrimRightFunc(output, unicode.IsSpace), ""
|
||||||
|
}
|
||||||
|
clean = strings.TrimRightFunc(output[:idx], unicode.IsSpace)
|
||||||
|
block = strings.Trim(output[idx+len(bankSeparator):], "\n")
|
||||||
|
return clean, block
|
||||||
|
}
|
||||||
|
|
||||||
|
// hasBankSrcHan reports whether src contains a CJK ideograph in the exact [㐀-鿿] range (U+3400–U+9FFF)
|
||||||
|
// banknote.py checks — a zh→ru channel line whose src has no Han is malformed. Kept as the exact
|
||||||
|
// Python range (NOT unicode.Han, which is wider) for byte-faithful parity.
|
||||||
|
func hasBankSrcHan(src string) bool {
|
||||||
|
for _, r := range src {
|
||||||
|
if r >= 0x3400 && r <= 0x9FFF {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseBanknote parses the tab-delimited block (integration seam feeding §C evidence + telemetry).
|
||||||
|
// Tolerant of a TRUNCATED final line when truncatedGeneration is set (§B3-5): a short last line under
|
||||||
|
// truncation is flagged banknote_truncated, not counted as a parse fail. Any other malformed line
|
||||||
|
// (fewer than 2 fields, or a src with no Han) sets banknote_parse_fail. Deterministic, no time/rand.
|
||||||
|
func parseBanknote(block string, truncatedGeneration bool) ([]bankEntry, bankFlags) {
|
||||||
|
var entries []bankEntry
|
||||||
|
var flags bankFlags
|
||||||
|
if strings.TrimSpace(block) == "" {
|
||||||
|
return entries, flags
|
||||||
|
}
|
||||||
|
var lines []string
|
||||||
|
for _, ln := range strings.Split(block, "\n") {
|
||||||
|
if strings.TrimSpace(ln) != "" {
|
||||||
|
lines = append(lines, ln)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
bad := 0
|
||||||
|
for i, ln := range lines {
|
||||||
|
raw := bankFieldSplit.Split(strings.TrimSpace(ln), -1)
|
||||||
|
var parts []string
|
||||||
|
for _, p := range raw {
|
||||||
|
if p = strings.TrimSpace(p); p != "" {
|
||||||
|
parts = append(parts, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(parts) < 2 {
|
||||||
|
// A short LAST line under a truncated generation is a tolerated cut, not a failure.
|
||||||
|
if i == len(lines)-1 && truncatedGeneration {
|
||||||
|
flags.Truncated = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
bad++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
src, dst := parts[0], parts[1]
|
||||||
|
typ := "term"
|
||||||
|
if len(parts) >= 3 {
|
||||||
|
typ = strings.ToLower(parts[2])
|
||||||
|
}
|
||||||
|
if !bankTypeOK[typ] {
|
||||||
|
typ = "term"
|
||||||
|
}
|
||||||
|
if !hasBankSrcHan(src) {
|
||||||
|
bad++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
entries = append(entries, bankEntry{Src: src, Dst: dst, Type: typ})
|
||||||
|
}
|
||||||
|
flags.NLines = len(entries)
|
||||||
|
flags.ParseFail = bad > 0
|
||||||
|
return entries, flags
|
||||||
|
}
|
||||||
101
backend/internal/pipeline/banknote_test.go
Normal file
101
backend/internal/pipeline/banknote_test.go
Normal file
|
|
@ -0,0 +1,101 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// banknote_test.go pins the WS4 §4(д) parser invariants (byte-faithful port of exp16 banknote.py).
|
||||||
|
// The 95-line corpus pin (0 parse_fail / 0 truncated over 14 saved raw outputs) is the domain of the
|
||||||
|
// $0 reference eval/design11/ws4_banknote_verify.py; these unit tests cover the parser CONTRACT.
|
||||||
|
|
||||||
|
func TestSplitBanknoteNoSeparatorIsAllClean(t *testing.T) {
|
||||||
|
out := "Судзуки шёл по коридору. \n"
|
||||||
|
clean, block := splitBanknote(out)
|
||||||
|
if clean != "Судзуки шёл по коридору." || block != "" {
|
||||||
|
t.Fatalf("no-SEP: clean=%q block=%q, want the whole right-trimmed output and empty block", clean, block)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSplitBanknoteSlicesBlock(t *testing.T) {
|
||||||
|
out := "Перевод... последнее предложение.\n\n" + bankSeparator + "\n方源\tФан Юань\tname\n蛊师\tгу-мастер\ttitle"
|
||||||
|
clean, block := splitBanknote(out)
|
||||||
|
if !strings.HasSuffix(clean, "последнее предложение.") {
|
||||||
|
t.Fatalf("clean tail wrong: %q", clean)
|
||||||
|
}
|
||||||
|
if strings.Contains(clean, bankSeparator) || strings.Contains(clean, "方源") {
|
||||||
|
t.Fatalf("the banknote block leaked into the clean translation: %q", clean)
|
||||||
|
}
|
||||||
|
ents, flags := parseBanknote(block, false)
|
||||||
|
if len(ents) != 2 || flags.ParseFail || flags.Truncated || flags.NLines != 2 {
|
||||||
|
t.Fatalf("parse = %+v flags=%+v, want 2 clean entries", ents, flags)
|
||||||
|
}
|
||||||
|
if ents[0].Src != "方源" || ents[0].Dst != "Фан Юань" || ents[0].Type != "name" {
|
||||||
|
t.Fatalf("entry[0] = %+v", ents[0])
|
||||||
|
}
|
||||||
|
if ents[1].Type != "title" {
|
||||||
|
t.Fatalf("entry[1] type = %q, want title", ents[1].Type)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBanknoteTolerantFieldSplit(t *testing.T) {
|
||||||
|
// tab, ≥2 spaces, and pipe are all accepted delimiters (a model that emits spaces instead of a
|
||||||
|
// real TAB still parses). type falls back to "term" when absent or unknown.
|
||||||
|
block := "方源\tФан Юань\tname\n蛊师 гу-мастер title\n青茅山 | гора Цинмао | place\n古月\tГу Юэ"
|
||||||
|
ents, flags := parseBanknote(block, false)
|
||||||
|
if flags.ParseFail {
|
||||||
|
t.Fatalf("tolerant split should not fail: %+v", flags)
|
||||||
|
}
|
||||||
|
if len(ents) != 4 {
|
||||||
|
t.Fatalf("want 4 entries across tab/space/pipe delimiters, got %d: %+v", len(ents), ents)
|
||||||
|
}
|
||||||
|
if ents[3].Type != "term" { // no type field → default term
|
||||||
|
t.Fatalf("missing type must default to term, got %q", ents[3].Type)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBanknoteNonHanSrcIsBad(t *testing.T) {
|
||||||
|
// A zh→ru channel line whose src has no Han ideograph is malformed → parse_fail.
|
||||||
|
block := "方源\tФан Юань\tname\nRoseanne\tРозанна\tname"
|
||||||
|
ents, flags := parseBanknote(block, false)
|
||||||
|
if !flags.ParseFail {
|
||||||
|
t.Fatalf("a non-Han src must set parse_fail")
|
||||||
|
}
|
||||||
|
if len(ents) != 1 || ents[0].Src != "方源" {
|
||||||
|
t.Fatalf("only the Han-src line should survive, got %+v", ents)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBanknoteTruncationTolerated(t *testing.T) {
|
||||||
|
// The LAST line cut by generation length is tolerated (banknote_truncated), not a parse fail —
|
||||||
|
// but ONLY under truncated_generation; the same short line otherwise IS a parse fail.
|
||||||
|
block := "方源\tФан Юань\tname\n蛊" // last line has no dst
|
||||||
|
entsT, flagsT := parseBanknote(block, true)
|
||||||
|
if !flagsT.Truncated || flagsT.ParseFail || len(entsT) != 1 {
|
||||||
|
t.Fatalf("truncated=true: want 1 entry + truncated flag + no parse_fail, got %+v %+v", entsT, flagsT)
|
||||||
|
}
|
||||||
|
entsF, flagsF := parseBanknote(block, false)
|
||||||
|
if flagsF.Truncated || !flagsF.ParseFail || len(entsF) != 1 {
|
||||||
|
t.Fatalf("truncated=false: the short last line must be a parse_fail, got %+v %+v", entsF, flagsF)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBanknoteEmptyBlock(t *testing.T) {
|
||||||
|
ents, flags := parseBanknote("", false)
|
||||||
|
if len(ents) != 0 || flags.ParseFail || flags.Truncated || flags.NLines != 0 {
|
||||||
|
t.Fatalf("empty block must yield no entries and clean flags, got %+v %+v", ents, flags)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBankSeparatorIsNotANoteWord(t *testing.T) {
|
||||||
|
// The separator must not be a trailing-note word the sanitizer reserves (§B3-1) — the two
|
||||||
|
// channels must never collide.
|
||||||
|
for _, note := range []string{"Примечание", "Сноска", "Комментарий", "Note", "TN"} {
|
||||||
|
if strings.Contains(bankSeparator, note) {
|
||||||
|
t.Fatalf("separator %q collides with the reserved note-word %q", bankSeparator, note)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(bankSeparator, "⟦") {
|
||||||
|
t.Fatalf("separator lost its unusual delimiter: %q", bankSeparator)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -97,6 +97,15 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// W0: eager-build every reachable client BEFORE any wave goroutine, so the clients map is
|
||||||
|
// read-only in the waves (r.client is lock-free, a miss is loud — closes the D12 lazy-init race).
|
||||||
|
if err := r.buildClients(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
// W0: build the per-model rate-guards (transport axis, read-only in the waves; a no-op until a
|
||||||
|
// model configures rate_limit — WS1 §1б, the mistral-arm precondition WS6).
|
||||||
|
r.buildRateGuards()
|
||||||
|
|
||||||
// Persist captured ruby readings (idempotent; consumed by seedGlossary below into
|
// Persist captured ruby readings (idempotent; consumed by seedGlossary below into
|
||||||
// auto glossary candidates — шаг 4). Never injected into a prompt here (§7d); a
|
// auto glossary candidates — шаг 4). Never injected into a prompt here (§7d); a
|
||||||
// resume re-persists the same rows at $0.
|
// resume re-persists the same rows at $0.
|
||||||
|
|
@ -122,7 +131,7 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
|
||||||
return nil, fmt.Errorf("pipeline: upsert snapshot %.12s: %w", snapID, err)
|
return nil, fmt.Errorf("pipeline: upsert snapshot %.12s: %w", snapID, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
chunks := SplitChunks(doc.Chapters)
|
chunks := SplitChunks(doc.Chapters, r.segBudget())
|
||||||
if len(chunks) == 0 {
|
if len(chunks) == 0 {
|
||||||
return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile)
|
return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,19 +7,21 @@ import (
|
||||||
|
|
||||||
// chunker.go: the шаг-3a SOURCE segmenter. It turns the per-chapter normalized
|
// chunker.go: the шаг-3a SOURCE segmenter. It turns the per-chapter normalized
|
||||||
// text produced by ingest.go into an ordered list of (chapter, chunk) units for
|
// text produced by ingest.go into an ordered list of (chapter, chunk) units for
|
||||||
// the runner loop. It replaced the Веха-2 placeholder (chapter split on \f +
|
// the runner loop. WS2 (слой 1) moved the packing budget to OUTPUT (ru) tokens via
|
||||||
// blank-line paragraph packing to a CHAR budget) with real segmentation: pack
|
// the fertility estimate (est_out = f_cjk·cjk + f_other·other over source char-classes),
|
||||||
// whole paragraphs to a ~1–2k-TOKEN budget, and when a single paragraph exceeds
|
// decoupling draft sizing (small chunk, COGS/coverage/alignment) from the EDIT unit
|
||||||
// the budget descend to SENTENCE boundaries — never splitting a sentence.
|
// (coarse chapter-scale unit for cross-chunk cohesion): pack whole paragraphs to
|
||||||
|
// DraftBudgetOut, and when a single paragraph exceeds it descend to SENTENCE boundaries
|
||||||
|
// (never splitting a sentence; a lone over-budget sentence is a passthrough
|
||||||
|
// OversizedSentence chunk). Edit units then group WHOLE draft chunks to EditCeilingOut.
|
||||||
//
|
//
|
||||||
// Contract kept intact for the loop (bookrun.go/chunkrun.go depend on it): SplitChunks emits
|
// Contract kept intact for the loop (bookrun.go/chunkrun.go depend on it): SplitChunks emits
|
||||||
// Chunk{Chapter,ChunkIdx,Text}; ChunkIdx resets to 0 per chapter; the function is
|
// Chunk{Chapter,ChunkIdx,Text,…}; ChunkIdx resets to 0 per chapter; the new fields (EstOut,
|
||||||
// a PURE, deterministic function of its input (no time/rand, no map iteration).
|
// OversizedSentence, EditUnitID) are ADDITIVE; the function is a PURE, deterministic function of
|
||||||
// Its behavior is versioned by chunkerVersion (render.go), folded into the job
|
// (chapters, SegBudget) (no time/rand, no map iteration). Its behavior is versioned by
|
||||||
// snapshot — so changing the segmentation is a loud --resnapshot re-translation,
|
// chunkerVersion (render.go); the SegBudget is snapshot-folded via segmentationSnap — so changing
|
||||||
// never a silent cache miss (§3.4/R6/D5.2). The chunk target is a CONST covered by
|
// the segmentation OR the budget/fertility is a loud --resnapshot re-translation, never a silent
|
||||||
// that version (Р2 = code); were it ever made config-tunable it MUST be folded into
|
// cache miss (§3.4/R6/D5.2/D30.9).
|
||||||
// the top-level snapshot like coverageSnap (§7d), not left to the version string.
|
|
||||||
//
|
//
|
||||||
// IMPORTANT: this sentence splitter is a SEPARATE, independent piece of logic from
|
// IMPORTANT: this sentence splitter is a SEPARATE, independent piece of logic from
|
||||||
// coverage.go's splitSentences. That one is a bit-for-bit port of the Python oracle
|
// coverage.go's splitSentences. That one is a bit-for-bit port of the Python oracle
|
||||||
|
|
@ -33,84 +35,113 @@ type Chunk struct {
|
||||||
Chapter int // 1-based
|
Chapter int // 1-based
|
||||||
ChunkIdx int // 0-based within its chapter
|
ChunkIdx int // 0-based within its chapter
|
||||||
Text string
|
Text string
|
||||||
|
// EstOut is the fertility-estimated OUTPUT (ru) token cost of this chunk (WS2, слой 1):
|
||||||
|
// est_out = fert.CJK·cjk_src + fert.Other·other_src over the source char-classes. It is the
|
||||||
|
// unit the draft budget packs against (DraftBudgetOut), decoupling draft sizing from the
|
||||||
|
// old input-token heuristic (L2-budget-wrong-unit). Additive field; the {Chapter,ChunkIdx,Text}
|
||||||
|
// contract the loop/ingest/status depend on is untouched.
|
||||||
|
EstOut float64
|
||||||
|
// OversizedSentence marks a chunk that is a SINGLE source sentence whose est_out alone exceeds
|
||||||
|
// DraftBudgetOut (WS2 §2б). It is passthrough (never clause-split — clause splitting is
|
||||||
|
// boundary machinery, anti-scope), and the flag makes the un-budgetable unit OBSERVABLE
|
||||||
|
// ("деградация не молчит") rather than silently oversized.
|
||||||
|
OversizedSentence bool
|
||||||
|
// EditUnitID is the 0-based book-global id of the coarse EDIT unit this draft chunk belongs to
|
||||||
|
// (WS2 decoupling: draft = small chunk, edit = large unit). An edit unit is a greedy grouping of
|
||||||
|
// WHOLE draft chunks up to EditCeilingOut (per chapter), so W2's editor reads a unit's draft as
|
||||||
|
// the concatenation of its member chunks' W1 outputs — clean reconstruction by construction (no
|
||||||
|
// straddle; verified: grouping whole chunks reproduces the paragraph-packed count 37 with 0
|
||||||
|
// straddle on the rerun corpus). Draft chunks never cross an edit-unit boundary.
|
||||||
|
EditUnitID int
|
||||||
}
|
}
|
||||||
|
|
||||||
// targetChunkTokens is the packing budget: paragraphs (or, inside an oversize
|
// SegBudget is the resolved output-token segmentation budget (WS2 §2в): the draft-chunk and
|
||||||
// paragraph, sentences) are greedily packed until the WHOLE chunk's estimated
|
// edit-unit ceilings in ru-OUTPUT tokens, plus the per-pair fertility coefficients that convert
|
||||||
// tokens (EstimateTokens' formula, applied once — see appendChapterChunks) would
|
// source char-classes to an output-token estimate. Resolved from config.Segmentation by the runner
|
||||||
// exceed this. ~1500 sits in the plan's "1–2k токенов по границам абзацев" corridor
|
// and snapshot-folded via segmentationSnap, so a budget/fertility edit is a loud --resnapshot
|
||||||
// (02-mvp Фаза 1); a chunk of ordinary prose lands well above the coverage gate's
|
// (D30.9). The classifier (tokenClassCounts) uses the REAL unicode.RangeTable of EstimateTokens
|
||||||
// 500-char floor, so the excision gate is not silently disabled (§7b). A chunk can
|
// (Han|Hiragana|Katakana|Hangul) for generality (§0.1) — NOT script-specific ord ranges.
|
||||||
// still fall under the floor only when a whole chapter (or its tail) is genuinely
|
type SegBudget struct {
|
||||||
// that short — a documented boundary. A const, not config: the size is a code
|
DraftBudgetOut float64 // draft-chunk ceiling, ru-output tokens (default 1797)
|
||||||
// segmentation rule (Р2), covered by chunkerVersion.
|
EditCeilingOut float64 // edit-unit ceiling, ru-output tokens (default 3200, gated arm at 8000)
|
||||||
const targetChunkTokens = 1500
|
FertCJK float64 // output tokens per CJK source char (default 1.1978)
|
||||||
|
FertOther float64 // output tokens per non-CJK/non-space source char (default 0.3852)
|
||||||
|
}
|
||||||
|
|
||||||
|
// estOut applies the fertility formula to pre-counted source char-classes — identical to
|
||||||
|
// est_out over the joined chunk text (whitespace is class-free, as tokenClassCounts drops it).
|
||||||
|
func (s SegBudget) estOut(cjk, other int) float64 {
|
||||||
|
return s.FertCJK*float64(cjk) + s.FertOther*float64(other)
|
||||||
|
}
|
||||||
|
|
||||||
// SplitChunks segments the ordered, per-chapter NORMALIZED text of a Document
|
// SplitChunks segments the ordered, per-chapter NORMALIZED text of a Document
|
||||||
// (ingest.go already applied NormalizeSource and split chapters) into an ordered
|
// (ingest.go already applied NormalizeSource and split chapters) into an ordered
|
||||||
// chunk list. Chapters that yield no text (blank/whitespace-only, e.g. an epub
|
// chunk list. Chapters that yield no text (blank/whitespace-only, e.g. an epub
|
||||||
// cover/nav page) do NOT consume a chapter number, so numbering stays dense over
|
// cover/nav page) do NOT consume a chapter number, so numbering stays dense over
|
||||||
// real content (Фаза-0 backward compat). Fully deterministic.
|
// real content (Фаза-0 backward compat). Fully deterministic. Each chunk carries its
|
||||||
func SplitChunks(chapters []string) []Chunk {
|
// EstOut + OversizedSentence flag + EditUnitID (WS2); the edit-unit id is monotone across
|
||||||
|
// the book so a W2 unit is uniquely addressable.
|
||||||
|
func SplitChunks(chapters []string, seg SegBudget) []Chunk {
|
||||||
var out []Chunk
|
var out []Chunk
|
||||||
chapterNo := 0
|
chapterNo := 0
|
||||||
|
editUnitID := 0
|
||||||
for _, chapText := range chapters {
|
for _, chapText := range chapters {
|
||||||
paras := splitParagraphs(chapText)
|
paras := splitParagraphs(chapText)
|
||||||
if len(paras) == 0 {
|
if len(paras) == 0 {
|
||||||
continue // an empty chapter does not consume a chapter number
|
continue // an empty chapter does not consume a chapter number
|
||||||
}
|
}
|
||||||
chapterNo++
|
chapterNo++
|
||||||
out = appendChapterChunks(out, chapterNo, paras)
|
chapterChunks := chapterDraftChunks(chapterNo, paras, seg)
|
||||||
|
assignEditUnits(chapterChunks, seg, &editUnitID)
|
||||||
|
out = append(out, chapterChunks...)
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
// appendChapterChunks packs one chapter's paragraphs into chunks and appends them.
|
// chapterDraftChunks packs one chapter's paragraphs into DRAFT chunks (the fine tiling). Rule:
|
||||||
// Rule: pack whole paragraphs while the running token estimate stays within
|
// pack whole paragraphs while the running OUTPUT-token estimate stays within DraftBudgetOut
|
||||||
// targetChunkTokens (prefer paragraph boundaries); a paragraph that alone exceeds
|
// (prefer paragraph boundaries); a paragraph that alone exceeds the budget is flushed on its own
|
||||||
// the budget is flushed on its own and then SENTENCE-packed (never splitting a
|
// and then SENTENCE-packed (never splitting a sentence; a single sentence over budget is its own
|
||||||
// sentence; a single sentence over budget is its own chunk).
|
// OversizedSentence chunk). The running budget is tracked as ADDITIVE char-class counts (CJK vs
|
||||||
//
|
// other), applying est_out ONCE to the whole chunk — the "\n\n" separators are whitespace, which
|
||||||
// The running budget is tracked as ADDITIVE char-class counts (CJK vs other),
|
// tokenClassCounts drops, so counting classes and estimating once reproduces est_out(joined text)
|
||||||
// applying EstimateTokens' formula (and its 16-token floor) ONCE to the whole
|
// exactly. Each emitted chunk gets its ChunkIdx (per-chapter) + EstOut; EditUnitID is set later.
|
||||||
// chunk — NOT by summing per-unit EstimateTokens. Summing per-unit floors every
|
func chapterDraftChunks(chapterNo int, paras []string, seg SegBudget) []Chunk {
|
||||||
// tiny paragraph up to 16 tokens, so a dialogue chapter of many short lines would
|
var out []Chunk
|
||||||
// flush a chunk at ~⅓ its true token content, needlessly dropping real content
|
|
||||||
// under the coverage gate's 500-char floor (self-review finding). Counting classes
|
|
||||||
// and flooring once reproduces EstimateTokens(joined text) exactly (the "\n\n"
|
|
||||||
// separators are whitespace, which EstimateTokens ignores).
|
|
||||||
func appendChapterChunks(out []Chunk, chapterNo int, paras []string) []Chunk {
|
|
||||||
chunkIdx := 0
|
chunkIdx := 0
|
||||||
|
emit := func(text string, oversized bool, cjk, other int) {
|
||||||
|
if t := strings.TrimSpace(text); t != "" {
|
||||||
|
out = append(out, Chunk{
|
||||||
|
Chapter: chapterNo, ChunkIdx: chunkIdx, Text: t,
|
||||||
|
EstOut: seg.estOut(cjk, other), OversizedSentence: oversized,
|
||||||
|
})
|
||||||
|
chunkIdx++
|
||||||
|
}
|
||||||
|
}
|
||||||
var buf []string // paragraphs accumulated for the current chunk
|
var buf []string // paragraphs accumulated for the current chunk
|
||||||
var bufCJK, bufOther int
|
var bufCJK, bufOther int
|
||||||
flush := func() {
|
flush := func() {
|
||||||
if len(buf) == 0 {
|
if len(buf) == 0 {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if text := strings.TrimSpace(strings.Join(buf, "\n\n")); text != "" {
|
emit(strings.Join(buf, "\n\n"), false, bufCJK, bufOther)
|
||||||
out = append(out, Chunk{Chapter: chapterNo, ChunkIdx: chunkIdx, Text: text})
|
|
||||||
chunkIdx++
|
|
||||||
}
|
|
||||||
buf = buf[:0]
|
buf = buf[:0]
|
||||||
bufCJK, bufOther = 0, 0
|
bufCJK, bufOther = 0, 0
|
||||||
}
|
}
|
||||||
for _, p := range paras {
|
for _, p := range paras {
|
||||||
pCJK, pOther := tokenClassCounts(p)
|
pCJK, pOther := tokenClassCounts(p)
|
||||||
if estTokensFrom(pCJK, pOther) > targetChunkTokens {
|
if seg.estOut(pCJK, pOther) > seg.DraftBudgetOut {
|
||||||
// Oversize paragraph: close any open chunk at this paragraph boundary,
|
// Oversize paragraph: close any open chunk at this paragraph boundary,
|
||||||
// then split the paragraph at sentence boundaries into its own chunks.
|
// then split the paragraph at sentence boundaries into its own chunks.
|
||||||
flush()
|
flush()
|
||||||
for _, sub := range packSentences(p) {
|
for _, sub := range packSentences(p, seg) {
|
||||||
if text := strings.TrimSpace(sub); text != "" {
|
emit(sub.text, sub.oversizedSentence, sub.cjk, sub.other)
|
||||||
out = append(out, Chunk{Chapter: chapterNo, ChunkIdx: chunkIdx, Text: text})
|
|
||||||
chunkIdx++
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
// Normal paragraph: start a new chunk if adding it would exceed the budget
|
// Normal paragraph: start a new chunk if adding it would exceed the budget
|
||||||
// and the current chunk is non-empty (never split a paragraph here).
|
// and the current chunk is non-empty (never split a paragraph here).
|
||||||
if len(buf) > 0 && estTokensFrom(bufCJK+pCJK, bufOther+pOther) > targetChunkTokens {
|
if len(buf) > 0 && seg.estOut(bufCJK+pCJK, bufOther+pOther) > seg.DraftBudgetOut {
|
||||||
flush()
|
flush()
|
||||||
}
|
}
|
||||||
buf = append(buf, p)
|
buf = append(buf, p)
|
||||||
|
|
@ -120,28 +151,67 @@ func appendChapterChunks(out []Chunk, chapterNo int, paras []string) []Chunk {
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
// packSentences splits one oversize paragraph into sentence segments (lossless
|
// assignEditUnits groups a chapter's already-built DRAFT chunks into EDIT units — greedy packing of
|
||||||
// tiling — concatenation reproduces the paragraph) and packs consecutive segments
|
// WHOLE draft chunks until the accumulated est_out would exceed EditCeilingOut — and stamps each
|
||||||
// into ≤ targetChunkTokens sub-chunks, never splitting a sentence. A single
|
// chunk's EditUnitID (monotone across the book via *nextID). Because units are formed from whole
|
||||||
// sentence larger than the budget becomes its own (oversize) sub-chunk — the only
|
// draft chunks, a unit's source/draft span is exactly the concatenation of its member chunks (no
|
||||||
// way to honor "never split a sentence". Returns the sub-chunk texts in order. Uses
|
// straddle) and W2 reconstructs the unit's draft losslessly. A single draft chunk larger than the
|
||||||
// the same additive char-class budget as appendChapterChunks (floor applied once).
|
// ceiling (an OversizedSentence, or a chapter whose first chunk already exceeds the ceiling) is its
|
||||||
func packSentences(paragraph string) []string {
|
// own unit — grouping never splits a chunk (never-split-below-chunk mirrors never-split-paragraph).
|
||||||
|
func assignEditUnits(chunks []Chunk, seg SegBudget, nextID *int) {
|
||||||
|
acc := 0.0
|
||||||
|
started := false
|
||||||
|
for i := range chunks {
|
||||||
|
if started && acc+chunks[i].EstOut > seg.EditCeilingOut {
|
||||||
|
*nextID++ // close the current unit at this chunk boundary
|
||||||
|
acc = 0
|
||||||
|
}
|
||||||
|
chunks[i].EditUnitID = *nextID
|
||||||
|
acc += chunks[i].EstOut
|
||||||
|
started = true
|
||||||
|
}
|
||||||
|
if started {
|
||||||
|
*nextID++ // advance past the last unit so the next chapter starts a fresh id
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// subChunk is one sentence-packed sub-chunk with its char-class counts and the oversized-sentence
|
||||||
|
// flag (a single sentence whose est_out alone exceeds DraftBudgetOut).
|
||||||
|
type subChunk struct {
|
||||||
|
text string
|
||||||
|
cjk, other int
|
||||||
|
oversizedSentence bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// packSentences splits one oversize paragraph into sentence segments (lossless tiling —
|
||||||
|
// concatenation reproduces the paragraph) and packs consecutive segments into ≤ DraftBudgetOut
|
||||||
|
// (est_out) sub-chunks, never splitting a sentence. A single sentence larger than the budget
|
||||||
|
// becomes its own sub-chunk flagged OversizedSentence (passthrough, never clause-split — the plan's
|
||||||
|
// §2б decision: "деградация не молчит" is satisfied by the FLAG, not emergency clause machinery).
|
||||||
|
func packSentences(paragraph string, seg SegBudget) []subChunk {
|
||||||
segs := splitSourceSentences(paragraph)
|
segs := splitSourceSentences(paragraph)
|
||||||
var chunks []string
|
var chunks []subChunk
|
||||||
var buf strings.Builder
|
var buf strings.Builder
|
||||||
var bufCJK, bufOther int
|
var bufCJK, bufOther int
|
||||||
flush := func() {
|
flush := func() {
|
||||||
if buf.Len() == 0 {
|
if buf.Len() == 0 {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
chunks = append(chunks, buf.String())
|
chunks = append(chunks, subChunk{text: buf.String(), cjk: bufCJK, other: bufOther})
|
||||||
buf.Reset()
|
buf.Reset()
|
||||||
bufCJK, bufOther = 0, 0
|
bufCJK, bufOther = 0, 0
|
||||||
}
|
}
|
||||||
for _, s := range segs {
|
for _, s := range segs {
|
||||||
sCJK, sOther := tokenClassCounts(s)
|
sCJK, sOther := tokenClassCounts(s)
|
||||||
if buf.Len() > 0 && estTokensFrom(bufCJK+sCJK, bufOther+sOther) > targetChunkTokens {
|
// A lone sentence over budget: flush what precedes it, then emit it as its own
|
||||||
|
// OversizedSentence sub-chunk (never split a sentence, and never merge it with a
|
||||||
|
// neighbour so the oversized flag stays attributable to the single sentence).
|
||||||
|
if seg.estOut(sCJK, sOther) > seg.DraftBudgetOut {
|
||||||
|
flush()
|
||||||
|
chunks = append(chunks, subChunk{text: s, cjk: sCJK, other: sOther, oversizedSentence: true})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if buf.Len() > 0 && seg.estOut(bufCJK+sCJK, bufOther+sOther) > seg.DraftBudgetOut {
|
||||||
flush()
|
flush()
|
||||||
}
|
}
|
||||||
buf.WriteString(s)
|
buf.WriteString(s)
|
||||||
|
|
@ -170,16 +240,6 @@ func tokenClassCounts(s string) (cjk, other int) {
|
||||||
return cjk, other
|
return cjk, other
|
||||||
}
|
}
|
||||||
|
|
||||||
// estTokensFrom applies EstimateTokens' formula (cjk + other/3, floored at 16) to
|
|
||||||
// pre-counted class totals — identical to EstimateTokens over the joined text.
|
|
||||||
func estTokensFrom(cjk, other int) int {
|
|
||||||
est := cjk + other/3
|
|
||||||
if est < 16 {
|
|
||||||
est = 16
|
|
||||||
}
|
|
||||||
return est
|
|
||||||
}
|
|
||||||
|
|
||||||
// splitParagraphs breaks a chapter into paragraphs on blank lines and trims each,
|
// splitParagraphs breaks a chapter into paragraphs on blank lines and trims each,
|
||||||
// dropping empties. Deterministic; whitespace-only paragraphs vanish. (Ingest emits
|
// dropping empties. Deterministic; whitespace-only paragraphs vanish. (Ingest emits
|
||||||
// a blank line between block-level epub elements, so paragraphs survive extraction.)
|
// a blank line between block-level epub elements, so paragraphs survive extraction.)
|
||||||
|
|
|
||||||
|
|
@ -8,43 +8,72 @@ import (
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// testSeg is the ratified WS2 output-token budget (zh-ru defaults) used by the chunker tests.
|
||||||
|
func testSeg() SegBudget {
|
||||||
|
return SegBudget{DraftBudgetOut: 1797, EditCeilingOut: 3200, FertCJK: 1.1978, FertOther: 0.3852}
|
||||||
|
}
|
||||||
|
|
||||||
|
// stripMeta zeroes the WS2 additive fields (EstOut/OversizedSentence/EditUnitID) so a
|
||||||
|
// {Chapter,ChunkIdx,Text}-level DeepEqual still expresses the intent of the pre-WS2 assertions.
|
||||||
|
func stripMeta(cs []Chunk) []Chunk {
|
||||||
|
out := make([]Chunk, len(cs))
|
||||||
|
for i, c := range cs {
|
||||||
|
out[i] = Chunk{Chapter: c.Chapter, ChunkIdx: c.ChunkIdx, Text: c.Text}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func TestSplitChunksSingleParagraph(t *testing.T) {
|
func TestSplitChunksSingleParagraph(t *testing.T) {
|
||||||
got := SplitChunks([]string{"静かな図書館の朝。"})
|
got := SplitChunks([]string{"静かな図書館の朝。"}, testSeg())
|
||||||
want := []Chunk{{Chapter: 1, ChunkIdx: 0, Text: "静かな図書館の朝。"}}
|
want := []Chunk{{Chapter: 1, ChunkIdx: 0, Text: "静かな図書館の朝。"}}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||||
t.Fatalf("single paragraph = %+v, want %+v", got, want)
|
t.Fatalf("single paragraph = %+v, want %+v", got, want)
|
||||||
}
|
}
|
||||||
|
// WS2: the additive fields are populated — est_out>0 and a single unit id.
|
||||||
|
if got[0].EstOut <= 0 || got[0].EditUnitID != 0 || got[0].OversizedSentence {
|
||||||
|
t.Fatalf("WS2 fields = est_out %.3f unit %d oversized %v", got[0].EstOut, got[0].EditUnitID, got[0].OversizedSentence)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSplitChunksChapters(t *testing.T) {
|
func TestSplitChunksChapters(t *testing.T) {
|
||||||
got := SplitChunks([]string{"Глава один.", "Глава два.", "Глава три."})
|
got := SplitChunks([]string{"Глава один.", "Глава два.", "Глава три."}, testSeg())
|
||||||
want := []Chunk{
|
want := []Chunk{
|
||||||
{Chapter: 1, ChunkIdx: 0, Text: "Глава один."},
|
{Chapter: 1, ChunkIdx: 0, Text: "Глава один."},
|
||||||
{Chapter: 2, ChunkIdx: 0, Text: "Глава два."},
|
{Chapter: 2, ChunkIdx: 0, Text: "Глава два."},
|
||||||
{Chapter: 3, ChunkIdx: 0, Text: "Глава три."},
|
{Chapter: 3, ChunkIdx: 0, Text: "Глава три."},
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||||
t.Fatalf("chapters = %+v, want %+v", got, want)
|
t.Fatalf("chapters = %+v, want %+v", got, want)
|
||||||
}
|
}
|
||||||
|
// WS2: each chapter is its own edit unit → distinct monotone EditUnitIDs 0,1,2.
|
||||||
|
for i, c := range got {
|
||||||
|
if c.EditUnitID != i {
|
||||||
|
t.Fatalf("chapter %d chunk edit_unit_id = %d, want %d (each short chapter is its own unit)", c.Chapter, c.EditUnitID, i)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSplitChunksPacksToTarget(t *testing.T) {
|
func TestSplitChunksPacksToTarget(t *testing.T) {
|
||||||
// Two paragraphs that together exceed the TOKEN target (each ~1000 tokens for
|
// Two paragraphs that together exceed the OUTPUT-token budget (each ~1156 out-tokens for
|
||||||
// cyrillic: 3000 chars / 3) must split into two chunks, each kept whole.
|
// cyrillic: 3000 chars × 0.3852) must split into two draft chunks, each kept whole.
|
||||||
p1 := strings.Repeat("а", 3000)
|
p1 := strings.Repeat("а", 3000)
|
||||||
p2 := strings.Repeat("б", 3000)
|
p2 := strings.Repeat("б", 3000)
|
||||||
got := SplitChunks([]string{p1 + "\n\n" + p2})
|
got := SplitChunks([]string{p1 + "\n\n" + p2}, testSeg())
|
||||||
want := []Chunk{
|
want := []Chunk{
|
||||||
{Chapter: 1, ChunkIdx: 0, Text: p1},
|
{Chapter: 1, ChunkIdx: 0, Text: p1},
|
||||||
{Chapter: 1, ChunkIdx: 1, Text: p2},
|
{Chapter: 1, ChunkIdx: 1, Text: p2},
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||||
t.Fatalf("packing = %d chunks, want 2 whole paragraphs; got %+v", len(got), summarize(got))
|
t.Fatalf("packing = %d chunks, want 2 whole paragraphs; got %+v", len(got), summarize(got))
|
||||||
}
|
}
|
||||||
|
// WS2: the two ~1156-out chunks (2311 together ≤ 3200) group into ONE edit unit.
|
||||||
|
if got[0].EditUnitID != got[1].EditUnitID {
|
||||||
|
t.Fatalf("two sub-ceiling draft chunks must share an edit unit, got ids %d and %d", got[0].EditUnitID, got[1].EditUnitID)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSplitChunksPacksSmallParagraphsTogether(t *testing.T) {
|
func TestSplitChunksPacksSmallParagraphsTogether(t *testing.T) {
|
||||||
got := SplitChunks([]string{"Абзац один.\n\nАбзац два.\n\nАбзац три."})
|
got := SplitChunks([]string{"Абзац один.\n\nАбзац два.\n\nАбзац три."}, testSeg())
|
||||||
if len(got) != 1 {
|
if len(got) != 1 {
|
||||||
t.Fatalf("small paragraphs must pack into one chunk, got %d: %+v", len(got), summarize(got))
|
t.Fatalf("small paragraphs must pack into one chunk, got %d: %+v", len(got), summarize(got))
|
||||||
}
|
}
|
||||||
|
|
@ -54,17 +83,17 @@ func TestSplitChunksPacksSmallParagraphsTogether(t *testing.T) {
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSplitChunksOversizeParagraphSplitsAtSentences(t *testing.T) {
|
func TestSplitChunksOversizeParagraphSplitsAtSentences(t *testing.T) {
|
||||||
// One paragraph far over the token target, built of many whole sentences, must
|
// One paragraph far over the output budget, built of many whole sentences, must
|
||||||
// split into >1 chunk AND never cut a sentence: every original sentence must
|
// split into >1 chunk AND never cut a sentence: every original sentence must
|
||||||
// survive intact and in order across the chunks.
|
// survive intact and in order across the chunks.
|
||||||
sentence := strings.Repeat("слово ", 40) + "конец." // ~80 tokens each
|
sentence := strings.Repeat("слово ", 40) + "конец." // ~79 out-tokens each
|
||||||
var sb strings.Builder
|
var sb strings.Builder
|
||||||
for i := 0; i < 40; i++ { // ~3200 tokens total ≫ 1500
|
for i := 0; i < 40; i++ { // ~3160 out-tokens total ≫ 1797
|
||||||
sb.WriteString(sentence)
|
sb.WriteString(sentence)
|
||||||
sb.WriteByte(' ')
|
sb.WriteByte(' ')
|
||||||
}
|
}
|
||||||
para := strings.TrimSpace(sb.String())
|
para := strings.TrimSpace(sb.String())
|
||||||
got := SplitChunks([]string{para})
|
got := SplitChunks([]string{para}, testSeg())
|
||||||
if len(got) < 2 {
|
if len(got) < 2 {
|
||||||
t.Fatalf("oversize paragraph must split into >1 chunk, got %d", len(got))
|
t.Fatalf("oversize paragraph must split into >1 chunk, got %d", len(got))
|
||||||
}
|
}
|
||||||
|
|
@ -79,41 +108,43 @@ func TestSplitChunksOversizeParagraphSplitsAtSentences(t *testing.T) {
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSplitChunksShortLinesPackByTrueTokenBudget(t *testing.T) {
|
func TestSplitChunksShortLinesPackByTrueTokenBudget(t *testing.T) {
|
||||||
// Many short dialogue paragraphs must pack by TRUE token content, not by summing
|
// Many short dialogue paragraphs must pack by TRUE output-token content (est_out on the
|
||||||
// the per-unit 16-token floor. 150 lines of 「はい。」 are ~450 true tokens (cjk=300,
|
// joined text), not by summing a per-unit floor. 150 lines of 「はい。」 are ~533 out-tokens,
|
||||||
// other=450 → 300+150), so they belong in ONE chunk. Summing the floor (150×16)
|
// so they belong in ONE chunk (a per-unit floor would wrongly flush into sub-500-char pieces).
|
||||||
// would wrongly flush into ~2 sub-500-char chunks the coverage gate then skips.
|
|
||||||
var paras []string
|
var paras []string
|
||||||
for i := 0; i < 150; i++ {
|
for i := 0; i < 150; i++ {
|
||||||
paras = append(paras, "「はい。」")
|
paras = append(paras, "「はい。」")
|
||||||
}
|
}
|
||||||
got := SplitChunks([]string{strings.Join(paras, "\n\n")})
|
got := SplitChunks([]string{strings.Join(paras, "\n\n")}, testSeg())
|
||||||
if len(got) != 1 {
|
if len(got) != 1 {
|
||||||
t.Fatalf("short lines must pack by true token budget into 1 chunk, got %d chunks", len(got))
|
t.Fatalf("short lines must pack by true token budget into 1 chunk, got %d chunks", len(got))
|
||||||
}
|
}
|
||||||
if EstimateTokens(got[0].Text) > targetChunkTokens {
|
if got[0].EstOut > testSeg().DraftBudgetOut {
|
||||||
t.Fatalf("packed chunk overshoots the budget: %d tokens", EstimateTokens(got[0].Text))
|
t.Fatalf("packed chunk overshoots the budget: est_out %.1f", got[0].EstOut)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSplitChunksOversizeSentenceIsOwnChunk(t *testing.T) {
|
func TestSplitChunksOversizeSentenceIsOwnChunk(t *testing.T) {
|
||||||
// A single sentence (no internal terminator) larger than the budget can only be
|
// A single sentence (no internal terminator) larger than the budget can only be its own
|
||||||
// its own chunk — never split.
|
// chunk — never split — and is flagged OversizedSentence (passthrough, WS2 §2б).
|
||||||
big := strings.Repeat("ы", 6000) // ~2000 tokens, one un-terminated run
|
big := strings.Repeat("ы", 6000) // ~2311 out-tokens, one un-terminated run
|
||||||
got := SplitChunks([]string{big})
|
got := SplitChunks([]string{big}, testSeg())
|
||||||
if len(got) != 1 || got[0].Text != big {
|
if len(got) != 1 || got[0].Text != big {
|
||||||
t.Fatalf("an oversize sentence must be one un-split chunk, got %+v", summarize(got))
|
t.Fatalf("an oversize sentence must be one un-split chunk, got %+v", summarize(got))
|
||||||
}
|
}
|
||||||
|
if !got[0].OversizedSentence {
|
||||||
|
t.Fatalf("a single over-budget sentence must be flagged OversizedSentence")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSplitChunksDropsEmpties(t *testing.T) {
|
func TestSplitChunksDropsEmpties(t *testing.T) {
|
||||||
// A blank/whitespace-only chapter vanishes and does NOT consume a chapter number.
|
// A blank/whitespace-only chapter vanishes and does NOT consume a chapter number.
|
||||||
got := SplitChunks([]string{"A", " \n\n ", "B"})
|
got := SplitChunks([]string{"A", " \n\n ", "B"}, testSeg())
|
||||||
want := []Chunk{
|
want := []Chunk{
|
||||||
{Chapter: 1, ChunkIdx: 0, Text: "A"},
|
{Chapter: 1, ChunkIdx: 0, Text: "A"},
|
||||||
{Chapter: 2, ChunkIdx: 0, Text: "B"},
|
{Chapter: 2, ChunkIdx: 0, Text: "B"},
|
||||||
}
|
}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(stripMeta(got), want) {
|
||||||
t.Fatalf("empties = %+v, want %+v", got, want)
|
t.Fatalf("empties = %+v, want %+v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -123,7 +154,7 @@ func TestSplitChunksDeterministic(t *testing.T) {
|
||||||
"Абзац.\n\nЕщё абзац.",
|
"Абзац.\n\nЕщё абзац.",
|
||||||
strings.Repeat("слово ", 400) + "конец.",
|
strings.Repeat("слово ", 400) + "конец.",
|
||||||
}
|
}
|
||||||
a, b := SplitChunks(chapters), SplitChunks(chapters)
|
a, b := SplitChunks(chapters, testSeg()), SplitChunks(chapters, testSeg())
|
||||||
if !reflect.DeepEqual(a, b) {
|
if !reflect.DeepEqual(a, b) {
|
||||||
t.Fatal("SplitChunks must be deterministic")
|
t.Fatal("SplitChunks must be deterministic")
|
||||||
}
|
}
|
||||||
|
|
@ -133,7 +164,7 @@ func TestSplitChunksNeverSplitsAbbreviation(t *testing.T) {
|
||||||
// Two long en paragraphs each ending mid-flow with "Mr. Smith" must not chunk
|
// Two long en paragraphs each ending mid-flow with "Mr. Smith" must not chunk
|
||||||
// between "Mr." and "Smith" — the abbreviation guard keeps the sentence whole.
|
// between "Mr." and "Smith" — the abbreviation guard keeps the sentence whole.
|
||||||
p := strings.Repeat("The road went on and on. ", 100) + "It was Mr. Smith who arrived."
|
p := strings.Repeat("The road went on and on. ", 100) + "It was Mr. Smith who arrived."
|
||||||
got := SplitChunks([]string{p})
|
got := SplitChunks([]string{p}, testSeg())
|
||||||
for _, c := range got {
|
for _, c := range got {
|
||||||
if strings.HasSuffix(strings.TrimSpace(c.Text), "Mr.") {
|
if strings.HasSuffix(strings.TrimSpace(c.Text), "Mr.") {
|
||||||
t.Fatalf("chunk ended at an abbreviation 'Mr.' — a sentence was cut: %q", c.Text)
|
t.Fatalf("chunk ended at an abbreviation 'Mr.' — a sentence was cut: %q", c.Text)
|
||||||
|
|
@ -141,6 +172,32 @@ func TestSplitChunksNeverSplitsAbbreviation(t *testing.T) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestSplitChunksEditUnitGroupsWholeChunks pins the WS2 decoupling: a chapter whose draft chunks
|
||||||
|
// together exceed EditCeilingOut is grouped into >1 edit unit at DRAFT-CHUNK boundaries (never
|
||||||
|
// splitting a chunk), so every chunk of a unit reconstructs the unit's draft losslessly.
|
||||||
|
func TestSplitChunksEditUnitGroupsWholeChunks(t *testing.T) {
|
||||||
|
// Three ~1156-out paragraphs (0.3852×3000): draft chunks {p1},{p2},{p3} (each ~1156, two
|
||||||
|
// together 2311 ≤ 1797? no — 2311>1797 so each is its own draft chunk). Edit ceiling 3200:
|
||||||
|
// group whole chunks → {p1,p2}=2311 ≤3200, +p3=3467>3200 → unit0={p1,p2}, unit1={p3}.
|
||||||
|
p := strings.Repeat("а", 3000)
|
||||||
|
q := strings.Repeat("б", 3000)
|
||||||
|
r := strings.Repeat("в", 3000)
|
||||||
|
got := SplitChunks([]string{p + "\n\n" + q + "\n\n" + r}, testSeg())
|
||||||
|
if len(got) != 3 {
|
||||||
|
t.Fatalf("want 3 draft chunks, got %d", len(got))
|
||||||
|
}
|
||||||
|
ids := []int{got[0].EditUnitID, got[1].EditUnitID, got[2].EditUnitID}
|
||||||
|
if ids[0] != ids[1] || ids[1] == ids[2] {
|
||||||
|
t.Fatalf("edit-unit grouping = %v, want first two together and the third separate", ids)
|
||||||
|
}
|
||||||
|
// Every draft chunk belongs to exactly one unit; a unit's chunks are contiguous.
|
||||||
|
for i := 1; i < len(got); i++ {
|
||||||
|
if got[i].EditUnitID < got[i-1].EditUnitID {
|
||||||
|
t.Fatalf("edit unit ids must be monotone, got %v", ids)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// --- source sentence splitter (zh/ja/en) ---------------------------------------
|
// --- source sentence splitter (zh/ja/en) ---------------------------------------
|
||||||
|
|
||||||
func TestSplitSourceSentencesTilesLosslessly(t *testing.T) {
|
func TestSplitSourceSentencesTilesLosslessly(t *testing.T) {
|
||||||
|
|
@ -262,7 +319,7 @@ func FuzzSplitChunksPreservesContent(f *testing.F) {
|
||||||
norm := NormalizeSource(s)
|
norm := NormalizeSource(s)
|
||||||
want := countNonSpace(norm)
|
want := countNonSpace(norm)
|
||||||
got := 0
|
got := 0
|
||||||
for _, c := range SplitChunks([]string{norm}) {
|
for _, c := range SplitChunks([]string{norm}, testSeg()) {
|
||||||
got += countNonSpace(c.Text)
|
got += countNonSpace(c.Text)
|
||||||
}
|
}
|
||||||
if got != want {
|
if got != want {
|
||||||
|
|
|
||||||
|
|
@ -226,7 +226,7 @@ func TestIngestEPUBRubyChapterMatchesDenseNumbering(t *testing.T) {
|
||||||
if len(doc.Ruby) != 1 || doc.Ruby[0].Chapter != 2 {
|
if len(doc.Ruby) != 1 || doc.Ruby[0].Chapter != 2 {
|
||||||
t.Fatalf("ruby dense chapter = %#v, want chapter 2", doc.Ruby)
|
t.Fatalf("ruby dense chapter = %#v, want chapter 2", doc.Ruby)
|
||||||
}
|
}
|
||||||
chunks := SplitChunks(doc.Chapters)
|
chunks := SplitChunks(doc.Chapters, testSeg())
|
||||||
var rubyChunkChapter int
|
var rubyChunkChapter int
|
||||||
for _, c := range chunks {
|
for _, c := range chunks {
|
||||||
if strings.Contains(c.Text, "朱雀") {
|
if strings.Contains(c.Text, "朱雀") {
|
||||||
|
|
|
||||||
|
|
@ -129,7 +129,15 @@ type MemoryBank struct {
|
||||||
entries []memoryEntry
|
entries []memoryEntry
|
||||||
ac *ahoCorasick
|
ac *ahoCorasick
|
||||||
keyOwners map[string][]int // normalized key → indices of entries that contributed it
|
keyOwners map[string][]int // normalized key → indices of entries that contributed it
|
||||||
version string // memoryVersion(): hash of frozen APPROVED rows + algo versions (F1/D8)
|
// enrichedVersion folds ALL foldable rows incl Source:mined — the memory version of the EDIT
|
||||||
|
// wave (snapshot_W2). baseVersion EXCLUDES Source:mined — the memory version of the DRAFT wave
|
||||||
|
// (snapshot_W1), so a W1.5 bank-enrichment (adding mined rows) moves ONLY snapshot_W2, keeping
|
||||||
|
// W1 checkpoints valid (WS1 §1в, «переоплата ОДНА»). The two are DOMAIN-SEPARATED (a "base-excl-
|
||||||
|
// mined" tag in the base fold), so base ≠ enriched for EVERY row set — even a mined-free book (a
|
||||||
|
// W1 job must never content-address to a W2 checkpoint). The load-bearing invariant is not
|
||||||
|
// equality but STABILITY: base stays fixed as mined rows are added; only enriched moves.
|
||||||
|
enrichedVersion string
|
||||||
|
baseVersion string
|
||||||
}
|
}
|
||||||
|
|
||||||
// pickedEntry is one selected record for a chunk with its firing key and disposition.
|
// pickedEntry is one selected record for a chunk with its firing key and disposition.
|
||||||
|
|
@ -170,7 +178,13 @@ type memorySelection struct {
|
||||||
activeIDs map[string]injectionDisposition
|
activeIDs map[string]injectionDisposition
|
||||||
}
|
}
|
||||||
|
|
||||||
func (b *MemoryBank) Version() string { return b.version }
|
// Version is the ENRICHED memory version (all approved rows incl mined) — the edit-wave / whole-book
|
||||||
|
// memory component. BaseVersion excludes Source:mined (the draft-wave component). They are
|
||||||
|
// DOMAIN-SEPARATED — base ≠ enriched for ANY row set, including a mined-free book (never assert
|
||||||
|
// equality) — but base stays STABLE when a W1.5 pass adds mined rows, which is what keeps W1
|
||||||
|
// checkpoints valid («переоплата ОДНА»).
|
||||||
|
func (b *MemoryBank) Version() string { return b.enrichedVersion }
|
||||||
|
func (b *MemoryBank) BaseVersion() string { return b.baseVersion }
|
||||||
|
|
||||||
// materializeMemory builds a MemoryBank from the book's stored glossary rows (ORDER
|
// materializeMemory builds a MemoryBank from the book's stored glossary rows (ORDER
|
||||||
// BY-stable — GlossaryForBook). Pure and deterministic. It computes memoryVersion as a
|
// BY-stable — GlossaryForBook). Pure and deterministic. It computes memoryVersion as a
|
||||||
|
|
@ -184,11 +198,11 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
|
||||||
|
|
||||||
for _, row := range rows {
|
for _, row := range rows {
|
||||||
e := memoryEntry{
|
e := memoryEntry{
|
||||||
id: row.Src + "\x1f" + row.Sense + "\x1f" + strconv.Itoa(row.SinceCh) + "\x1f" + strconv.Itoa(row.UntilCh),
|
id: row.Src + "\x1f" + row.Sense + "\x1f" + strconv.Itoa(row.SinceCh) + "\x1f" + strconv.Itoa(row.UntilCh),
|
||||||
src: row.Src,
|
src: row.Src,
|
||||||
dst: row.Dst,
|
dst: row.Dst,
|
||||||
status: row.Status,
|
status: row.Status,
|
||||||
sense: row.Sense,
|
sense: row.Sense,
|
||||||
sinceCh: row.SinceCh,
|
sinceCh: row.SinceCh,
|
||||||
untilCh: row.UntilCh,
|
untilCh: row.UntilCh,
|
||||||
allowShort: row.AllowShort,
|
allowShort: row.AllowShort,
|
||||||
|
|
@ -240,7 +254,8 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
|
||||||
}
|
}
|
||||||
sort.Strings(allKeys) // deterministic automaton construction
|
sort.Strings(allKeys) // deterministic automaton construction
|
||||||
b.ac = buildAC(allKeys)
|
b.ac = buildAC(allKeys)
|
||||||
b.version = computeMemoryVersion(rows, gateOn)
|
b.enrichedVersion = computeMemoryVersion(rows, gateOn) // all approved incl mined (W2)
|
||||||
|
b.baseVersion = computeMemoryVersionScoped(rows, gateOn, true) // excl Source:mined (W1)
|
||||||
return b
|
return b
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -256,14 +271,31 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
|
||||||
// glossary.id autoincrement is deliberately EXCLUDED (fresh each replace); only content
|
// glossary.id autoincrement is deliberately EXCLUDED (fresh each replace); only content
|
||||||
// columns are hashed.
|
// columns are hashed.
|
||||||
func computeMemoryVersion(rows []store.GlossaryEntry, gateOn bool) string {
|
func computeMemoryVersion(rows []store.GlossaryEntry, gateOn bool) string {
|
||||||
|
return computeMemoryVersionScoped(rows, gateOn, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// computeMemoryVersionScoped is computeMemoryVersion with an explicit Source-scope: excludeMined
|
||||||
|
// drops every Source:"mined" row from the fold (WS1 §1в base-bank-version). The DRAFT-wave snapshot
|
||||||
|
// (snapshot_W1) folds base (excludeMined=true) so a W1.5 pass that ADDS mined rows moves ONLY the
|
||||||
|
// enriched (edit-wave) version — keeping W1 checkpoints valid, «переоплата ОДНА». Deterministic (a
|
||||||
|
// Source filter over the same ORDER BY-stable rows). excludeMined=false is BYTE-IDENTICAL to the
|
||||||
|
// pre-split fold (no extra field), so the enriched hash / existing snapshots do not move; the
|
||||||
|
// excludeMined=true variant adds a domain separator so base and enriched never collide.
|
||||||
|
func computeMemoryVersionScoped(rows []store.GlossaryEntry, gateOn, excludeMined bool) string {
|
||||||
h := sha256.New()
|
h := sha256.New()
|
||||||
h.Write([]byte("tm-memory-v2\x00"))
|
h.Write([]byte("tm-memory-v2\x00"))
|
||||||
h.Write([]byte(memoryNormVersion + "\x00" + memoryMatchVersion + "\x00"))
|
h.Write([]byte(memoryNormVersion + "\x00" + memoryMatchVersion + "\x00"))
|
||||||
h.Write([]byte("gate:" + strconv.FormatBool(gateOn) + "\x00"))
|
h.Write([]byte("gate:" + strconv.FormatBool(gateOn) + "\x00"))
|
||||||
|
if excludeMined {
|
||||||
|
h.Write([]byte("base-excl-mined\x00")) // domain separator: base ≠ enriched even over identical rows
|
||||||
|
}
|
||||||
for _, r := range rows {
|
for _, r := range rows {
|
||||||
if r.Status != "approved" && !gateOn {
|
if r.Status != "approved" && !gateOn {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
if excludeMined && r.Source == "mined" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
// A fixed, length-prefixed field layout so no content can forge a boundary.
|
// A fixed, length-prefixed field layout so no content can forge a boundary.
|
||||||
writeField(h, r.Status)
|
writeField(h, r.Status)
|
||||||
writeField(h, r.Src)
|
writeField(h, r.Src)
|
||||||
|
|
@ -492,39 +524,53 @@ func renderGlossaryBlock(injected []pickedEntry) string {
|
||||||
return glossaryBlockHeader + "\n" + strings.Join(lines, "\n")
|
return glossaryBlockHeader + "\n" + strings.Join(lines, "\n")
|
||||||
}
|
}
|
||||||
|
|
||||||
// editorConstraintHeader introduces the editor's dst-constraint block. It gives the editor the
|
// renderFormatVersion versions the FORMAT of the role-injection renderers whose output is NOT
|
||||||
// approved TARGET renderings as consistency constraints — it must normalise any divergent rendering
|
// captured by memoryMatchVersion (a scope/matcher-ALGORITHM version, not a render-layout version):
|
||||||
// in the draft to these canonical forms (inflecting for context) and touch nothing else.
|
// today the editor constraint block's src→dst layout (WS2 §2в) and, later, the gender-constraint
|
||||||
const editorConstraintHeader = "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике эти сущности должны быть переданы ИМЕННО этими формами — приводи к ним любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):"
|
// render (WS5). A format change shifts the injected bytes → the wire, so it must be a loud
|
||||||
|
// --resnapshot; folding it as a dedicated snapshot component (snapshot.go) keeps that loudness a
|
||||||
|
// MECHANISM, not discipline — memoryMatchVersion (const memory.go) would not move on a render edit.
|
||||||
|
const renderFormatVersion = "renderfmt-v1-editor-src2dst"
|
||||||
|
|
||||||
// renderEditorConstraintBlock serializes the selected records' CONFIRMED dst renderings into the
|
// editorConstraintHeader introduces the editor's canonical-constraint block. It gives the BILINGUAL
|
||||||
// editor's injection. Unlike renderGlossaryBlock it emits the TARGET forms only — bare canonical
|
// editor (D30.1) the approved src→dst bindings as consistency constraints — it must render each
|
||||||
// Russian, not "src → dst". It is CONFIRMED-only on purpose: an AMBIGUOUS record is a candidate the
|
// listed source term with EXACTLY the paired canonical form (inflecting for context) and touch
|
||||||
// translator MAY have legitimately rejected, so forcing the editor to rewrite the draft toward it
|
// nothing else.
|
||||||
// would corrupt a correct translation (mirrors the gate's CONFIRMED-only discipline, external-review
|
const editorConstraintHeader = "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):"
|
||||||
// #1). NOTE (D30.1 supersede of D1): the editor is now BILINGUAL — editor.md feeds it the source, so
|
|
||||||
// the earlier "monolingual editor never sees the source" rationale for target-forms-only is stale
|
// renderEditorConstraintBlock serializes the selected records' CONFIRMED renderings into the
|
||||||
// (the CONFIRMED-only discipline above is the LIVE reason and is unchanged here). Whether a bilingual
|
// editor's injection as a src→dst MAPPING (WS2 §2в resolves the D30.1-open question): "源термин →
|
||||||
// editor should receive the src→dst mapping (like the translator) so it can bind each canonical form
|
// «dst»", like the translator block, NOT bare canonical Russian. Rationale (the single load-bearing
|
||||||
// to its source term, rather than bare Russian, is an OPEN design question the D30.1 flip left
|
// one, ревью-1 F4): the editor is BILINGUAL (editor.md feeds it the source), and binding the canon
|
||||||
// unclosed (flagged to the orchestrator; behaviour deliberately unchanged in this pack). Deduped by
|
// to its SOURCE term disambiguates HOMONYMIC dst (one Russian surface for two entities) — bare
|
||||||
// dst (an alias and its main entry share a rendering), in the budget priority order fixed by Select.
|
// forms cannot. It stays CONFIRMED-only on purpose: an AMBIGUOUS record is a candidate the
|
||||||
// Returns "" when nothing CONFIRMED renders → the editor gets NO injection (the plain draft-only
|
// translator MAY have legitimately rejected, so forcing the editor to rewrite toward it would
|
||||||
|
// corrupt a correct translation (mirrors the gate's CONFIRMED-only discipline, external-review #1);
|
||||||
|
// AMBIGUOUS/mined-draft rows never enter here. Deduped by (src,dst) — an alias and its main entry
|
||||||
|
// share the pair — which only WIDENS the list (harmless), in the budget priority order fixed by
|
||||||
|
// Select. Returns "" when nothing CONFIRMED renders → the editor gets NO injection (plain draft-only
|
||||||
// layout). The block is a subset of the already budget-limited memSel.injected, so it needs no
|
// layout). The block is a subset of the already budget-limited memSel.injected, so it needs no
|
||||||
// separate token budget.
|
// separate token budget. The src→dst FORMAT is snapshot-folded via renderFormatVersion (a format
|
||||||
|
// edit is a loud --resnapshot); the injection is a message, so it also enters request_hash
|
||||||
|
// directly (no silent false-hit).
|
||||||
func renderEditorConstraintBlock(injected []pickedEntry) string {
|
func renderEditorConstraintBlock(injected []pickedEntry) string {
|
||||||
var lines []string
|
var lines []string
|
||||||
seen := map[string]bool{}
|
seen := map[[2]string]bool{}
|
||||||
for _, p := range injected {
|
for _, p := range injected {
|
||||||
if p.disp != memConfirmed {
|
if p.disp != memConfirmed {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
src := strings.TrimSpace(p.entry.src)
|
||||||
dst := strings.TrimSpace(p.entry.dst)
|
dst := strings.TrimSpace(p.entry.dst)
|
||||||
if dst == "" || seen[dst] {
|
if dst == "" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
seen[dst] = true
|
key := [2]string{src, dst}
|
||||||
lines = append(lines, "- «"+dst+"»")
|
if seen[key] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = true
|
||||||
|
lines = append(lines, "- "+src+" → «"+dst+"»")
|
||||||
}
|
}
|
||||||
if len(lines) == 0 {
|
if len(lines) == 0 {
|
||||||
return ""
|
return ""
|
||||||
|
|
|
||||||
|
|
@ -529,36 +529,45 @@ func TestInjectivityCollision(t *testing.T) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRenderEditorConstraintBlock covers D1: the monolingual editor's injection is the
|
// TestRenderEditorConstraintBlock covers WS2 §2в (D30.1 open-question resolved): the BILINGUAL
|
||||||
// CONFIRMED dst forms ONLY (target constraints), deduped, with no source or "src → dst".
|
// editor's injection is a CONFIRMED src→dst MAPPING (binding each canon to its source term, so a
|
||||||
// Reverting the CONFIRMED-only guard leaks an AMBIGUOUS candidate and fails here.
|
// homonymic dst stays distinguishable), NOT bare target forms. It stays CONFIRMED-only (an
|
||||||
|
// AMBIGUOUS candidate must not force a rewrite); dedup is by (src,dst), which only widens the list.
|
||||||
func TestRenderEditorConstraintBlock(t *testing.T) {
|
func TestRenderEditorConstraintBlock(t *testing.T) {
|
||||||
suzuki := &memoryEntry{src: "鈴木", dst: "Судзуки", status: "approved"}
|
suzuki := &memoryEntry{src: "鈴木", dst: "Судзуки", status: "approved"}
|
||||||
cand := &memoryEntry{src: "小D", dst: "Малыш Дэ", status: "auto"} // AMBIGUOUS
|
cand := &memoryEntry{src: "小D", dst: "Малыш Дэ", status: "auto"} // AMBIGUOUS
|
||||||
tanaka := &memoryEntry{src: "田中", dst: "Танака", status: "approved"}
|
tanaka := &memoryEntry{src: "田中", dst: "Танака", status: "approved"}
|
||||||
tanakaKana := &memoryEntry{src: "たなか", dst: "Танака", status: "approved"} // same dst → deduped
|
tanakaDup := &memoryEntry{src: "田中", dst: "Танака", status: "approved"} // same (src,dst) → deduped
|
||||||
|
// A distinct source term that renders to the SAME Russian surface — the homonym case the
|
||||||
|
// src→dst binding exists for: both lines survive (widening), the editor can tell them apart.
|
||||||
|
homonym := &memoryEntry{src: "済", dst: "Танака", status: "approved"}
|
||||||
sel := []pickedEntry{
|
sel := []pickedEntry{
|
||||||
{entry: suzuki, via: "鈴木", disp: memConfirmed},
|
{entry: suzuki, via: "鈴木", disp: memConfirmed},
|
||||||
{entry: cand, via: "小d", disp: memAmbiguous},
|
{entry: cand, via: "小d", disp: memAmbiguous},
|
||||||
{entry: tanaka, via: "田中", disp: memConfirmed},
|
{entry: tanaka, via: "田中", disp: memConfirmed},
|
||||||
{entry: tanakaKana, via: "たなか", disp: memConfirmed},
|
{entry: tanakaDup, via: "田中", disp: memConfirmed},
|
||||||
|
{entry: homonym, via: "済", disp: memConfirmed},
|
||||||
}
|
}
|
||||||
block := renderEditorConstraintBlock(sel)
|
block := renderEditorConstraintBlock(sel)
|
||||||
|
|
||||||
if !strings.Contains(block, "«Судзуки»") || !strings.Contains(block, "«Танака»") {
|
// src→dst mapping present under the editor's own header.
|
||||||
t.Errorf("confirmed dst forms missing: %q", block)
|
if !strings.Contains(block, "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ") {
|
||||||
|
t.Errorf("editor header missing: %q", block)
|
||||||
|
}
|
||||||
|
if !strings.Contains(block, "鈴木 → «Судзуки»") || !strings.Contains(block, "田中 → «Танака»") {
|
||||||
|
t.Errorf("src→dst mapping missing: %q", block)
|
||||||
}
|
}
|
||||||
// AMBIGUOUS excluded — the editor must not be forced toward an unverified rendering.
|
// AMBIGUOUS excluded — the editor must not be forced toward an unverified rendering.
|
||||||
if strings.Contains(block, "Малыш Дэ") {
|
if strings.Contains(block, "Малыш Дэ") || strings.Contains(block, "小D") {
|
||||||
t.Errorf("AMBIGUOUS candidate leaked into the editor constraints: %q", block)
|
t.Errorf("AMBIGUOUS candidate leaked into the editor constraints: %q", block)
|
||||||
}
|
}
|
||||||
// dst-only: no source key, no arrow (the editor is monolingual).
|
// Deduped by (src,dst): 田中→Танака appears once despite the duplicate entry.
|
||||||
if strings.Contains(block, "→") || strings.Contains(block, "鈴木") || strings.Contains(block, "ГЛОССАРИЙ") {
|
if n := strings.Count(block, "田中 → «Танака»"); n != 1 {
|
||||||
t.Errorf("editor block must be target-only, no src→dst: %q", block)
|
t.Errorf("(src,dst) not deduped: 田中→Танака appears %d times: %q", n, block)
|
||||||
}
|
}
|
||||||
// Deduped by dst (Танака appears once despite two source keys).
|
// Homonym widening: a DIFFERENT source with the same dst is NOT collapsed — both lines survive.
|
||||||
if n := strings.Count(block, "«Танака»"); n != 1 {
|
if !strings.Contains(block, "済 → «Танака»") {
|
||||||
t.Errorf("dst not deduped: «Танака» appears %d times: %q", n, block)
|
t.Errorf("homonymic dst must stay distinguishable by source term: %q", block)
|
||||||
}
|
}
|
||||||
// An all-AMBIGUOUS (or empty) selection yields no block at all.
|
// An all-AMBIGUOUS (or empty) selection yields no block at all.
|
||||||
if got := renderEditorConstraintBlock([]pickedEntry{{entry: cand, disp: memAmbiguous}}); got != "" {
|
if got := renderEditorConstraintBlock([]pickedEntry{{entry: cand, disp: memAmbiguous}}); got != "" {
|
||||||
|
|
|
||||||
38
backend/internal/pipeline/memory_wave_test.go
Normal file
38
backend/internal/pipeline/memory_wave_test.go
Normal file
|
|
@ -0,0 +1,38 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestBaseEnrichedMemoryVersionSplit pins the WS1 §1в per-wave bank-version mechanism: the BASE
|
||||||
|
// version (draft wave / snapshot_W1) excludes Source:"mined" rows, so adding a mined approved row
|
||||||
|
// after W1 moves ONLY the ENRICHED version (edit wave / snapshot_W2) — keeping W1 checkpoints valid
|
||||||
|
// («переоплата ОДНА»). A broken split (base folding mined) would move snapshot_W1 and re-bill W1.
|
||||||
|
func TestBaseEnrichedMemoryVersionSplit(t *testing.T) {
|
||||||
|
seedRows := []store.GlossaryEntry{
|
||||||
|
{Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"},
|
||||||
|
{Src: "古月", Dst: "Гу Юэ", Status: "approved", Source: "ruby"},
|
||||||
|
}
|
||||||
|
base0 := materializeMemory(seedRows, false)
|
||||||
|
// Same rows PLUS a mined approved row (as a W1.5 pass would add).
|
||||||
|
enrichedRows := append([]store.GlossaryEntry{}, seedRows...)
|
||||||
|
enrichedRows = append(enrichedRows, store.GlossaryEntry{Src: "蛊", Dst: "гу", Status: "approved", Source: "mined"})
|
||||||
|
bank1 := materializeMemory(enrichedRows, false)
|
||||||
|
|
||||||
|
// 1) Base version is UNCHANGED by the mined addition (draft wave stays valid).
|
||||||
|
if base0.BaseVersion() != bank1.BaseVersion() {
|
||||||
|
t.Fatalf("base version moved on a mined-row addition — snapshot_W1 would re-bill W1:\n before %s\n after %s",
|
||||||
|
base0.BaseVersion(), bank1.BaseVersion())
|
||||||
|
}
|
||||||
|
// 2) Enriched version DID move (edit wave sees the new mined canon).
|
||||||
|
if base0.Version() == bank1.Version() {
|
||||||
|
t.Fatalf("enriched version did NOT move on a mined-row addition — the edit wave would miss the mined canon")
|
||||||
|
}
|
||||||
|
// 3) Base ≠ enriched even before any mined row (domain-separated), so a W1 job can never
|
||||||
|
// accidentally content-address to a W2 checkpoint.
|
||||||
|
if base0.BaseVersion() == base0.Version() {
|
||||||
|
t.Fatalf("base and enriched versions collide over identical rows — they must be domain-separated")
|
||||||
|
}
|
||||||
|
}
|
||||||
97
backend/internal/pipeline/ratelimit.go
Normal file
97
backend/internal/pipeline/ratelimit.go
Normal file
|
|
@ -0,0 +1,97 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ratelimit.go: the per-model rate-guard for the parallel waves (WS1 §1б, ревью-2 F4). The per-CALL
|
||||||
|
// 429/Retry-After backoff already lives in the transport adapter (httpllm.go); the gap the waves
|
||||||
|
// open is N-CONCURRENCY against a rate-limited model — mistral-large-latest fails ~48% of calls under
|
||||||
|
// parallelism (a token-bucket / concurrency cap, tier-dependent — 00-provider-quirks §Транспорт),
|
||||||
|
// while grok is 0%. A model caps its own wave concurrency (max_concurrency) and, optionally, paces
|
||||||
|
// call STARTS (min_interval). This is a TRANSPORT axis: it never touches the request bytes, so it is
|
||||||
|
// wire-neutral and NOT snapshot-folded. Guards are built once in W0, read-only in the waves; each
|
||||||
|
// guard's internals (a buffered-channel semaphore + a mutex-protected pacer) are concurrency-safe,
|
||||||
|
// so acquiring from N wave goroutines is race-clean.
|
||||||
|
|
||||||
|
// rateGuard limits concurrent calls to one model (+ optional pacing). The zero/nil guard is a no-op
|
||||||
|
// (unlimited), so an unconfigured model — or a nil map entry — costs nothing.
|
||||||
|
type rateGuard struct {
|
||||||
|
sem chan struct{} // buffered to max_concurrency; nil = unlimited
|
||||||
|
minInterval time.Duration // 0 = no pacing
|
||||||
|
mu sync.Mutex // guards last
|
||||||
|
last time.Time // start time of the most recent acquired call (pacer)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newRateGuard builds a guard. maxConcurrency ≤ 0 → unlimited concurrency; minInterval ≤ 0 → no
|
||||||
|
// pacing. Both off → newRateGuard returns a guard that acquire treats as a pure no-op.
|
||||||
|
func newRateGuard(maxConcurrency int, minInterval time.Duration) *rateGuard {
|
||||||
|
g := &rateGuard{minInterval: minInterval}
|
||||||
|
if maxConcurrency > 0 {
|
||||||
|
g.sem = make(chan struct{}, maxConcurrency)
|
||||||
|
}
|
||||||
|
return g
|
||||||
|
}
|
||||||
|
|
||||||
|
// acquire blocks until a concurrency slot is free and, if pacing is on, until the min-interval since
|
||||||
|
// the last acquired call has elapsed — both respecting ctx cancellation. It returns a release func
|
||||||
|
// the caller MUST defer (a no-op when acquire failed or the guard is unlimited). A nil guard (no
|
||||||
|
// config) acquires instantly. The pacer holds the mutex only to read/stamp `last`; the wait itself
|
||||||
|
// is outside the lock, so a paced model still admits one caller per interval without serializing the
|
||||||
|
// lock for the whole wait.
|
||||||
|
func (g *rateGuard) acquire(ctx context.Context) (func(), error) {
|
||||||
|
if g == nil {
|
||||||
|
return func() {}, nil
|
||||||
|
}
|
||||||
|
if g.sem != nil {
|
||||||
|
select {
|
||||||
|
case g.sem <- struct{}{}:
|
||||||
|
case <-ctx.Done():
|
||||||
|
return func() {}, ctx.Err()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
release := func() {
|
||||||
|
if g.sem != nil {
|
||||||
|
<-g.sem
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if g.minInterval > 0 {
|
||||||
|
for {
|
||||||
|
g.mu.Lock()
|
||||||
|
wait := time.Until(g.last.Add(g.minInterval))
|
||||||
|
if wait <= 0 {
|
||||||
|
g.last = time.Now()
|
||||||
|
g.mu.Unlock()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
g.mu.Unlock()
|
||||||
|
select {
|
||||||
|
case <-time.After(wait):
|
||||||
|
// re-check: another caller may have taken the slot while we slept.
|
||||||
|
case <-ctx.Done():
|
||||||
|
release()
|
||||||
|
return func() {}, ctx.Err()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return release, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildRateGuards constructs a guard for every reachable model in W0 (read-only in the waves).
|
||||||
|
// A model with no rate_limit config gets an unlimited (no-op) guard, so rateGuard(model) is always
|
||||||
|
// non-nil and lookup-safe.
|
||||||
|
func (r *Runner) buildRateGuards() {
|
||||||
|
r.rateGuards = make(map[string]*rateGuard, len(r.Models.Models))
|
||||||
|
for _, m := range r.reachableModels() {
|
||||||
|
rl := r.Models.Models[m].RateLimit
|
||||||
|
r.rateGuards[m] = newRateGuard(rl.MaxConcurrency, time.Duration(rl.MinIntervalMS)*time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// rateGuard returns the pre-built guard for a model (nil-safe: acquire treats a missing/nil guard as
|
||||||
|
// unlimited, so a model reached before buildRateGuards ran simply is not throttled).
|
||||||
|
func (r *Runner) rateGuard(model string) *rateGuard {
|
||||||
|
return r.rateGuards[model]
|
||||||
|
}
|
||||||
113
backend/internal/pipeline/ratelimit_test.go
Normal file
113
backend/internal/pipeline/ratelimit_test.go
Normal file
|
|
@ -0,0 +1,113 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestRateGuardLimitsConcurrency proves the semaphore REALLY bounds concurrency (WS6 precondition,
|
||||||
|
// ревью-2 F4): with max_concurrency=2, no more than 2 goroutines are ever inside the guarded region
|
||||||
|
// at once, even under 20 racing callers. A broken guard (missing/over-sized semaphore) trips the
|
||||||
|
// observed-max assertion.
|
||||||
|
func TestRateGuardLimitsConcurrency(t *testing.T) {
|
||||||
|
const cap, workers = 2, 20
|
||||||
|
g := newRateGuard(cap, 0)
|
||||||
|
var inFlight, maxSeen int64
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < workers; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
rel, err := g.acquire(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("acquire: %v", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
cur := atomic.AddInt64(&inFlight, 1)
|
||||||
|
for {
|
||||||
|
m := atomic.LoadInt64(&maxSeen)
|
||||||
|
if cur <= m || atomic.CompareAndSwapInt64(&maxSeen, m, cur) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(time.Millisecond) // widen the window a concurrency bug would exploit
|
||||||
|
atomic.AddInt64(&inFlight, -1)
|
||||||
|
rel()
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if maxSeen > cap {
|
||||||
|
t.Fatalf("rate-guard admitted %d concurrent callers, cap is %d", maxSeen, cap)
|
||||||
|
}
|
||||||
|
if maxSeen == 0 {
|
||||||
|
t.Fatalf("no caller was observed in-flight — the guard never admitted anyone")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRateGuardNilAndUnlimitedAreNoOps: a nil guard and a zero-config guard both admit instantly and
|
||||||
|
// never block (the sequential path / an unconfigured model must cost nothing).
|
||||||
|
func TestRateGuardNilAndUnlimitedAreNoOps(t *testing.T) {
|
||||||
|
var nilGuard *rateGuard
|
||||||
|
rel, err := nilGuard.acquire(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("nil guard acquire: %v", err)
|
||||||
|
}
|
||||||
|
rel() // must not panic
|
||||||
|
unlimited := newRateGuard(0, 0)
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
rel, err := unlimited.acquire(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unlimited acquire %d: %v", i, err)
|
||||||
|
}
|
||||||
|
rel()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRateGuardHonoursContextCancel: a caller blocked on a full semaphore unblocks on ctx cancel and
|
||||||
|
// does NOT leak a slot (the wave dispatcher must be able to abort a run cleanly).
|
||||||
|
func TestRateGuardHonoursContextCancel(t *testing.T) {
|
||||||
|
g := newRateGuard(1, 0)
|
||||||
|
rel, err := g.acquire(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first acquire: %v", err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
r, err := g.acquire(ctx) // blocks: the one slot is held
|
||||||
|
r()
|
||||||
|
done <- err
|
||||||
|
}()
|
||||||
|
cancel()
|
||||||
|
select {
|
||||||
|
case err := <-done:
|
||||||
|
if err != context.Canceled {
|
||||||
|
t.Fatalf("blocked acquire on cancel = %v, want context.Canceled", err)
|
||||||
|
}
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("acquire did not unblock on ctx cancel")
|
||||||
|
}
|
||||||
|
rel() // release the first; a fresh acquire must now succeed (no leaked slot)
|
||||||
|
r2, err := g.acquire(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("acquire after release: %v", err)
|
||||||
|
}
|
||||||
|
r2()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRateGuardPacingSpacesStarts: with min_interval set, two serial acquires are spaced by at least
|
||||||
|
// the interval (best-effort pacing of call STARTS).
|
||||||
|
func TestRateGuardPacingSpacesStarts(t *testing.T) {
|
||||||
|
g := newRateGuard(0, 30*time.Millisecond)
|
||||||
|
start := time.Now()
|
||||||
|
r1, _ := g.acquire(context.Background())
|
||||||
|
r1()
|
||||||
|
r2, _ := g.acquire(context.Background())
|
||||||
|
r2()
|
||||||
|
if elapsed := time.Since(start); elapsed < 30*time.Millisecond {
|
||||||
|
t.Fatalf("paced acquires took %v, want ≥30ms between starts", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -43,7 +43,11 @@ import (
|
||||||
// now normalized once in NormalizeSource (#3), the ASCII sentence boundary honors
|
// now normalized once in NormalizeSource (#3), the ASCII sentence boundary honors
|
||||||
// quote-depth (#4), and <br> inside <ruby> no longer leaks a newline into the body
|
// quote-depth (#4), and <br> inside <ruby> no longer leaks a newline into the body
|
||||||
// (#5) — all segmentation/ingest behaviour changes → a loud --resnapshot.
|
// (#5) — all segmentation/ingest behaviour changes → a loud --resnapshot.
|
||||||
const chunkerVersion = "chunker-v4-utf8-quotedepth"
|
// Bumped v4→v5 (WS2): the packing budget moved from input-token heuristic (const 1500) to OUTPUT
|
||||||
|
// (ru) tokens via fertility (est_out), draft chunks decoupled from coarse EDIT units (grouped whole
|
||||||
|
// chunks to EditCeilingOut), oversized-sentence flag added. The budget itself is ALSO folded via
|
||||||
|
// segmentationSnap (snapshot.go), so this version covers only the algorithm shape.
|
||||||
|
const chunkerVersion = "chunker-v5-output-budget-editunit"
|
||||||
|
|
||||||
// estimatorVersion versions EstimateTokens: его выход входит в max_tokens и
|
// estimatorVersion versions EstimateTokens: его выход входит в max_tokens и
|
||||||
// через него в request-hash, поэтому перекалибровка весов — тоже явная
|
// через него в request-hash, поэтому перекалибровка весов — тоже явная
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,7 @@ package pipeline
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
|
|
||||||
|
|
@ -43,6 +44,9 @@ type Runner struct {
|
||||||
|
|
||||||
clients map[string]llm.LLMClient
|
clients map[string]llm.LLMClient
|
||||||
templates map[string]*PromptTemplate
|
templates map[string]*PromptTemplate
|
||||||
|
// rateGuards is the per-model wave-concurrency guard set (WS1 §1б), built once in W0 and
|
||||||
|
// read-only in the waves — a transport axis, never snapshot-folded. nil until buildRateGuards.
|
||||||
|
rateGuards map[string]*rateGuard
|
||||||
|
|
||||||
// memory is the book's glossary FROZEN for this job (materialized once in
|
// memory is the book's glossary FROZEN for this job (materialized once in
|
||||||
// TranslateBook, after seeding, before snapshotID). Its Version() is the F1
|
// TranslateBook, after seeding, before snapshotID). Its Version() is the F1
|
||||||
|
|
@ -168,14 +172,63 @@ func (r *Runner) loadTemplates() error {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// segBudget resolves the pipeline's config.Segmentation into the pipeline-package SegBudget the
|
||||||
|
// chunker consumes (WS2). Kept in the runner (not chunker.go) so the segmenter stays config-free.
|
||||||
|
func (r *Runner) segBudget() SegBudget {
|
||||||
|
seg := r.Pipeline.Segmentation
|
||||||
|
return SegBudget{
|
||||||
|
DraftBudgetOut: float64(seg.DraftBudgetOut),
|
||||||
|
EditCeilingOut: float64(seg.EditCeilingOut),
|
||||||
|
FertCJK: seg.Fertility.CJK,
|
||||||
|
FertOther: seg.Fertility.Other,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// reachableModels enumerates every model a run can call: the union of the stage models and their
|
||||||
|
// single-hop escalate_to fallbacks (stagerun.go calls r.client ONLY with model∈{st.Model, EscalateTo}).
|
||||||
|
// This set is complete and static for a book, which is what lets W0 pre-build every client (a third
|
||||||
|
// model axis — channel B / annotator — must extend this to stay race-free). Deterministic order.
|
||||||
|
func (r *Runner) reachableModels() []string {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
var out []string
|
||||||
|
add := func(m string) {
|
||||||
|
if m != "" && !seen[m] {
|
||||||
|
seen[m] = true
|
||||||
|
out = append(out, m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, st := range r.Pipeline.Stages {
|
||||||
|
add(st.Model)
|
||||||
|
add(st.EscalateTo)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildClients EAGER-constructs every reachable LLM client BEFORE any wave goroutine starts (W0).
|
||||||
|
// After this the clients map is READ-ONLY in the waves, so r.client is lock-free and a miss is a
|
||||||
|
// loud error, not a lazy build under a data race — closing the runner.go lazy-init race (D12) and
|
||||||
|
// tripwiring a future un-enumerated model axis. Idempotent; BuildClient needs only the provider
|
||||||
|
// config (not the API key), so eager-build is safe even on a key-less resume.
|
||||||
|
func (r *Runner) buildClients() error {
|
||||||
|
for _, m := range r.reachableModels() {
|
||||||
|
if _, ok := r.clients[m]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
c, err := BuildClient(r.Models, m, r.Log)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("pipeline: eager-build client for model %q (W0): %w", m, err)
|
||||||
|
}
|
||||||
|
r.clients[m] = c
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// client returns the PRE-BUILT client for a model. Read-only (no lazy build, no lock): buildClients
|
||||||
|
// constructed every reachable client in W0, so the map is only read in the waves; a miss means an
|
||||||
|
// un-enumerated model reached the wire and is a loud error, never a silent lazy build under a race.
|
||||||
func (r *Runner) client(model string) (llm.LLMClient, error) {
|
func (r *Runner) client(model string) (llm.LLMClient, error) {
|
||||||
if c, ok := r.clients[model]; ok {
|
if c, ok := r.clients[model]; ok {
|
||||||
return c, nil
|
return c, nil
|
||||||
}
|
}
|
||||||
c, err := BuildClient(r.Models, model, r.Log)
|
return nil, fmt.Errorf("pipeline: no pre-built client for model %q — W0 buildClients enumerates stage models ∪ escalate_to; a model outside that set reached the wire, add it to the eager set to keep the waves race-free", model)
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
r.clients[model] = c
|
|
||||||
return c, nil
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -89,9 +89,9 @@ func TestRunnerMemoryInjectionAndPostcheck(t *testing.T) {
|
||||||
if strings.Contains(msgs[2].Content, "ГЛОССАРИЙ") {
|
if strings.Contains(msgs[2].Content, "ГЛОССАРИЙ") {
|
||||||
t.Errorf("glossary leaked into the user (ch.Text) message: %q", msgs[2].Content)
|
t.Errorf("glossary leaked into the user (ch.Text) message: %q", msgs[2].Content)
|
||||||
}
|
}
|
||||||
// D1: the monolingual EDITOR now gets the CONFIRMED dst forms as target constraints — its
|
// D30.1/WS2 §2в: the BILINGUAL editor gets the CONFIRMED canon as a src→dst MAPPING under its
|
||||||
// OWN dst-constraint block (distinct header), NOT the translator's "src → dst" ГЛОССАРИЙ,
|
// OWN header (КАНОНИЧЕСКИЕ), binding each canonical form to its source term (homonym-safe) — NOT
|
||||||
// and never the source key inside the injection.
|
// the translator's ГЛОССАРИЙ header, and never an AMBIGUOUS row.
|
||||||
editorSaw := false
|
editorSaw := false
|
||||||
for _, b := range rec.all() {
|
for _, b := range rec.all() {
|
||||||
if !isEditBody(b) {
|
if !isEditBody(b) {
|
||||||
|
|
@ -101,11 +101,11 @@ func TestRunnerMemoryInjectionAndPostcheck(t *testing.T) {
|
||||||
if len(em) != 3 || em[1].Role != "system" {
|
if len(em) != 3 || em[1].Role != "system" {
|
||||||
t.Fatalf("editor must have 3 messages (system, dst-constraints, user), got %d: %+v", len(em), em)
|
t.Fatalf("editor must have 3 messages (system, dst-constraints, user), got %d: %+v", len(em), em)
|
||||||
}
|
}
|
||||||
if !strings.Contains(em[1].Content, "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ") || !strings.Contains(em[1].Content, "Судзуки") {
|
if !strings.Contains(em[1].Content, "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ") || !strings.Contains(em[1].Content, "鈴木 → «Судзуки»") {
|
||||||
t.Errorf("editor dst-constraint block missing/wrong: %q", em[1].Content)
|
t.Errorf("editor src→dst constraint block missing/wrong: %q", em[1].Content)
|
||||||
}
|
}
|
||||||
if strings.Contains(em[1].Content, "ГЛОССАРИЙ") || strings.Contains(em[1].Content, "→") || strings.Contains(em[1].Content, "鈴木") {
|
if strings.Contains(em[1].Content, "ГЛОССАРИЙ") {
|
||||||
t.Errorf("editor block must be dst-only (no ГЛОССАРИЙ header / arrow / source): %q", em[1].Content)
|
t.Errorf("editor block must use its own КАНОНИЧЕСКИЕ header, not the translator ГЛОССАРИЙ: %q", em[1].Content)
|
||||||
}
|
}
|
||||||
editorSaw = true
|
editorSaw = true
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -11,15 +11,36 @@ import (
|
||||||
// громкий --resnapshot (= переоплата книги, D15), никогда тихий false-hit.
|
// громкий --resnapshot (= переоплата книги, D15), никогда тихий false-hit.
|
||||||
|
|
||||||
// contextSnap freezes the context-assembly knobs (§3.8) inside the snapshot.
|
// contextSnap freezes the context-assembly knobs (§3.8) inside the snapshot.
|
||||||
// Fixed field order — it is part of a content hash.
|
// Fixed field order — it is part of a content hash. WS2 dropped STMDepth/OverlapTokens
|
||||||
|
// (dead carryover knobs — §2а), a structural change folded loudly (§8 manifest).
|
||||||
type contextSnap struct {
|
type contextSnap struct {
|
||||||
GlossaryInjection string `json:"glossary_injection"`
|
GlossaryInjection string `json:"glossary_injection"`
|
||||||
GlossaryTokenBudget int `json:"glossary_token_budget"`
|
GlossaryTokenBudget int `json:"glossary_token_budget"`
|
||||||
STMDepth int `json:"stm_depth"`
|
|
||||||
OverlapTokens int `json:"overlap_tokens"`
|
|
||||||
CacheTTL string `json:"cache_ttl"`
|
CacheTTL string `json:"cache_ttl"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// segmentationSnap freezes the WS2 output-token chunking budget (draft/edit ceilings + per-pair
|
||||||
|
// fertility) inside the snapshot, mirroring coverageSnap's discipline: a budget or fertility edit
|
||||||
|
// re-chunks the book (changes chunk boundaries → the wire), so it is a loud --resnapshot (D30.9),
|
||||||
|
// never a silent divergence. Unlike coverageSnap it is ALWAYS folded — segmentation always runs and
|
||||||
|
// its values are wire-determining. Fixed field order — part of a content hash.
|
||||||
|
type segmentationSnap struct {
|
||||||
|
DraftBudgetOut int `json:"draft_budget_out"`
|
||||||
|
EditCeilingOut int `json:"edit_ceiling_out"`
|
||||||
|
FertilityCJK float64 `json:"fertility_cjk"`
|
||||||
|
FertilityOther float64 `json:"fertility_other"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Runner) segmentationSnapshot() segmentationSnap {
|
||||||
|
seg := r.Pipeline.Segmentation
|
||||||
|
return segmentationSnap{
|
||||||
|
DraftBudgetOut: seg.DraftBudgetOut,
|
||||||
|
EditCeilingOut: seg.EditCeilingOut,
|
||||||
|
FertilityCJK: seg.Fertility.CJK,
|
||||||
|
FertilityOther: seg.Fertility.Other,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// coverageSnap freezes the excision coverage-gate config inside the snapshot, so
|
// coverageSnap freezes the excision coverage-gate config inside the snapshot, so
|
||||||
// enabling the gate OR tuning its thresholds is a loud --resnapshot, never a silent
|
// enabling the gate OR tuning its thresholds is a loud --resnapshot, never a silent
|
||||||
// mismatch between an old checkpoint's stored disposition and a changed gate. HONEST
|
// mismatch between an old checkpoint's stored disposition and a changed gate. HONEST
|
||||||
|
|
@ -182,6 +203,14 @@ func (r *Runner) snapshotID() (id, payload string, err error) {
|
||||||
// ваем ДО инъекции, чтобы смена budget'а инъекции не промахнулась мимо
|
// ваем ДО инъекции, чтобы смена budget'а инъекции не промахнулась мимо
|
||||||
// resnapshot-гейта (D5.2). Фикс-порядок полей — это content-hash.
|
// resnapshot-гейта (D5.2). Фикс-порядок полей — это content-hash.
|
||||||
ContextAssembly contextSnap `json:"context_assembly"`
|
ContextAssembly contextSnap `json:"context_assembly"`
|
||||||
|
// Segmentation — the WS2 output-token chunk/edit-unit budget + fertility (segmentationSnap).
|
||||||
|
// It changes chunk boundaries → the wire, so a budget/fertility edit is a loud --resnapshot.
|
||||||
|
Segmentation segmentationSnap `json:"segmentation"`
|
||||||
|
// RenderFormatVersion versions the FORMAT of the role-injection renderers not captured by
|
||||||
|
// memoryMatchVersion (a scope/matcher-algorithm version): the editor constraint block's
|
||||||
|
// src→dst layout (WS2 §2в) and the gender-constraint render (WS5). A format change shifts the
|
||||||
|
// injected bytes → the wire, so folding it separately keeps a flip a loud --resnapshot.
|
||||||
|
RenderFormatVersion string `json:"render_format_version"`
|
||||||
// MemoryVersion — content-hash детерминированно материализованной инъекти-
|
// MemoryVersion — content-hash детерминированно материализованной инъекти-
|
||||||
// руемой памяти (approved-глоссарий+резюме+series-bible). Не bump-счётчик
|
// руемой памяти (approved-глоссарий+резюме+series-bible). Не bump-счётчик
|
||||||
// (D8): забыть пересчитать нельзя, а забытый bump переоткрыл бы класс тихой
|
// (D8): забыть пересчитать нельзя, а забытый bump переоткрыл бы класс тихой
|
||||||
|
|
@ -222,15 +251,15 @@ func (r *Runner) snapshotID() (id, payload string, err error) {
|
||||||
ContextAssembly: contextSnap{
|
ContextAssembly: contextSnap{
|
||||||
GlossaryInjection: r.Pipeline.Context.GlossaryInjection,
|
GlossaryInjection: r.Pipeline.Context.GlossaryInjection,
|
||||||
GlossaryTokenBudget: r.Pipeline.Context.GlossaryTokenBudget,
|
GlossaryTokenBudget: r.Pipeline.Context.GlossaryTokenBudget,
|
||||||
STMDepth: r.Pipeline.Context.STMDepth,
|
|
||||||
OverlapTokens: r.Pipeline.Context.OverlapTokens,
|
|
||||||
CacheTTL: r.Pipeline.Context.CacheTTL,
|
CacheTTL: r.Pipeline.Context.CacheTTL,
|
||||||
},
|
},
|
||||||
MemoryVersion: r.memoryVersion(),
|
Segmentation: r.segmentationSnapshot(),
|
||||||
PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate,
|
RenderFormatVersion: renderFormatVersion,
|
||||||
Coverage: r.coverageSnapshot(),
|
MemoryVersion: r.memoryVersion(),
|
||||||
StyleCheckVersion: cheapGateVersion,
|
PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate,
|
||||||
Sanitizer: r.sanitizerSnapshot(),
|
Coverage: r.coverageSnapshot(),
|
||||||
|
StyleCheckVersion: cheapGateVersion,
|
||||||
|
Sanitizer: r.sanitizerSnapshot(),
|
||||||
}
|
}
|
||||||
for _, st := range r.Pipeline.Stages {
|
for _, st := range r.Pipeline.Stages {
|
||||||
ss := stageSnap{
|
ss := stageSnap{
|
||||||
|
|
|
||||||
|
|
@ -352,6 +352,18 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID
|
||||||
}
|
}
|
||||||
att.freshCall = true
|
att.freshCall = true
|
||||||
|
|
||||||
|
// Per-model rate-guard (WS1 §1б): a wave-concurrency cap (+ optional pacing) around the ONE
|
||||||
|
// expensive provider call, so a rate-limited model (mistral) does not 429-storm under N-parallel
|
||||||
|
// waves. A no-op for an unconfigured model / the sequential path. Acquire respects ctx; on cancel
|
||||||
|
// the reservation is released and the call surfaces as an infra failure (like a dead client).
|
||||||
|
relGuard, err := r.rateGuard(model).acquire(ctx)
|
||||||
|
if err != nil {
|
||||||
|
r.releaseReservation(ctx, resv)
|
||||||
|
r.setJobStatus(ctx, job.ID, "failed")
|
||||||
|
return att, fmt.Errorf("pipeline: rate-guard acquire for %s/ch%d/chunk%d/%s (model %s): %w", r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, model, err)
|
||||||
|
}
|
||||||
|
defer relGuard()
|
||||||
|
|
||||||
// «Вызов в полёте» обязан быть виден: между этой строкой и «attempt completed»
|
// «Вызов в полёте» обязан быть виден: между этой строкой и «attempt completed»
|
||||||
// могут пройти минуты (attempt_timeout × ретраи транспорта), и без неё «висит
|
// могут пройти минуты (attempt_timeout × ретраи транспорта), и без неё «висит
|
||||||
// провайдер» неотличимо от «умер процесс» — главная боль smoke-прогона №4.
|
// провайдер» неотличимо от «умер процесс» — главная боль smoke-прогона №4.
|
||||||
|
|
|
||||||
|
|
@ -141,7 +141,7 @@ func (r *Runner) bookChunks() ([]Chunk, error) {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
return SplitChunks(doc.Chapters), nil
|
return SplitChunks(doc.Chapters, r.segBudget()), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// chunkKey identifies a chunk positionally.
|
// chunkKey identifies a chunk positionally.
|
||||||
|
|
|
||||||
File diff suppressed because one or more lines are too long
|
|
@ -1,3 +1,5 @@
|
||||||
|
> **⚠ СТАТУС: ЧАСТИЧНО ОТРАБОТАН — Block A + ядра ПРИНЯТЫ И ЗАЛЕНДЕНЫ (19.07, D39.13).** Сдано: WS2 ✅ (чанкер output-бюджет + edit-единицы + src→dst + golden) · WS1a/b/c-инфра ✅ (eager-clients, rate-guard, base/enriched split) · WS4-core ✅ (банкнота-парсер) · WS6-частично (конфиг-тай). **Резидуал R1–R5** (волновой ИСПОЛНИТЕЛЬ · WS3-майнер · WS4-12-точек · WS5-чекеры · арм-конфиги) — точные дизайны в `backend/PACK11_REPORT.md` §Резидуал. **Промт ОСТАЁТСЯ АКТИВНЫМ:** сессия-продолжение стартует по нему + отчёту (читать ОБА; сданное не пере-строить — оно под тестами).
|
||||||
|
|
||||||
# Промт: сессия БЭКЕНД — единый пак по плану 11 (волновой раннер · чанкер/единицы · банк W1.5 · банкнота · чекеры · армы), 2026-07-19
|
# Промт: сессия БЭКЕНД — единый пак по плану 11 (волновой раннер · чанкер/единицы · банк W1.5 · банкнота · чекеры · армы), 2026-07-19
|
||||||
|
|
||||||
> **Место в очереди (пост-D39.12):** исследовательская программа арх-ресета (D39–D39.10) и дизайн-синтез
|
> **Место в очереди (пост-D39.12):** исследовательская программа арх-ресета (D39–D39.10) и дизайн-синтез
|
||||||
|
|
|
||||||
|
|
@ -4,11 +4,15 @@
|
||||||
> - **Фазы:** 0 ✅ · Ф1-инфра ✅ (D20–D28; golden = инвариант №8) · арка «качество-первым» D26→D38.5 ЗАКРЫТА (хроника — `archive/PROGRESS-2026-07-10-13.md`; итог: потолок в организации, не в моделях). **АРХ-РЕСЕТ D39 (13.07):** синк-аудит «сломанного телефона» (65 находок / 0 refuted — `architecture/08-sync-audit-ledger.md`) → целевая 7-слойная архитектура + фазовый план (`architecture/09-target-architecture.md`); инвариант общности §0.1 (любая книга/пара/структура); промпт-тема консолидирована в `architecture/10-prompt-architecture.md`.
|
> - **Фазы:** 0 ✅ · Ф1-инфра ✅ (D20–D28; golden = инвариант №8) · арка «качество-первым» D26→D38.5 ЗАКРЫТА (хроника — `archive/PROGRESS-2026-07-10-13.md`; итог: потолок в организации, не в моделях). **АРХ-РЕСЕТ D39 (13.07):** синк-аудит «сломанного телефона» (65 находок / 0 refuted — `architecture/08-sync-audit-ledger.md`) → целевая 7-слойная архитектура + фазовый план (`architecture/09-target-architecture.md`); инвариант общности §0.1 (любая книга/пара/структура); промпт-тема консолидирована в `architecture/10-prompt-architecture.md`.
|
||||||
> - **Трек A (build-now) ЗАКРЫТ:** пак-1 (D39.2, `d3f6b34`: trust-gated suppressor [терм-дрейф закрыт в КОДЕ] · export-contract нормализация · pair-сеам `prompts:{zh-ru}` fail-loud · реестр role→инъекция · target-гейты · quality-report) + пак-1.5+F1–F9 (D39.5, `dc30c7d`: `tmctl export` [manifest-join+drift-гард; полигон-экстракция впредь ТОЛЬКО через него] · fold-first санитайзер [HIGH-обход закрыт] · bounded глосс-whitelist · пример zh→ru). Адверсариал-долг 529 погашен (D39.4: 8 осей, 0 CRIT/1 HIGH/6 MED — всё в фикс-листах/ledger).
|
> - **Трек A (build-now) ЗАКРЫТ:** пак-1 (D39.2, `d3f6b34`: trust-gated suppressor [терм-дрейф закрыт в КОДЕ] · export-contract нормализация · pair-сеам `prompts:{zh-ru}` fail-loud · реестр role→инъекция · target-гейты · quality-report) + пак-1.5+F1–F9 (D39.5, `dc30c7d`: `tmctl export` [manifest-join+drift-гард; полигон-экстракция впредь ТОЛЬКО через него] · fold-first санитайзер [HIGH-обход закрыт] · bounded глосс-whitelist · пример zh→ru). Адверсариал-долг 529 погашен (D39.4: 8 осей, 0 CRIT/1 HIGH/6 MED — всё в фикс-листах/ledger).
|
||||||
> - **Трек B (ресёрч) ЗАЛЕНДЕН:** research/19 нарезка+когезия+контракт t/e+волны W0–W3 (D39.1; конфликт exp14↔research/18 разрешён: эмиссия≠окно≠связность) · research/20 банк-майнинг W1.5 (D39.6; детектор-лестница V-A/B/C, алиас-ярусы, banknote-v1, плагины P1–P6, §D-пре-рег; конфаунд инъекции ⇒ cold-start срез).
|
> - **Трек B (ресёрч) ЗАЛЕНДЕН:** research/19 нарезка+когезия+контракт t/e+волны W0–W3 (D39.1; конфликт exp14↔research/18 разрешён: эмиссия≠окно≠связность) · research/20 банк-майнинг W1.5 (D39.6; детектор-лестница V-A/B/C, алиас-ярусы, banknote-v1, плагины P1–P6, §D-пре-рег; конфаунд инъекции ⇒ cold-start срез).
|
||||||
> - **exp15 ЗАВЕРШЁН И ЗАЛЕНДЕН (D39.7, `fe34a16`):** «граница вредит» отозван (артефакт окна судьи, свинг +0.689 после фикса), все когезия-эффекты под шум-полом 0.126, «sequential нигде не лучше» (LOO-stable, НЕ ратифицированный зелёный свет волне), фертильность 1.20·cjk+0.39·other; $12.79/$15. **Слепое чтение ИСПОЛНЕНО (D39.8):** планка не пройдена никем, дефекты распределены по армам (читатель подтвердил floor-null; V2-брак = риг без гейтов, не carryover-рычаг) → битва качества пере-прогона = дефект-классы (时辰-юниты · числа b1 · gender-enforce · стих/аллюзии · регистр-лексикон), не нарезка. **Q4a ИСПОЛНЕН И ЗАЛЕНДЕН (D39.9, $1.32/$2):** верность НЕ покупается на translate-стадии (flash уже верен 25/0/27; pro ×3.67 не ратифицирован); смысл ломает glm-РЕДАКТОР (свап glm→mistral/deepseek-pro = приоритетный арм пере-прогона); exp14b-правила дефектны → $0-ре-аудит в exp16. **exp16 ИСПОЛНЕН И ЗАЛЕНДЕН (D39.10, $0.02/$10):** WHICH подтверждена (код-детектор V-C recall 0.97 held-out, $0) / WHAT опровергнута для ко-оккуренции → dst-канал = банкнота (идея владельца оправдана) + облако+Палладий; 9B-споттер не нужен (0.895 ≤ код); Z1 → облако; ре-аудит exp14b: ядро D38 держится, errata не нужна. Тачпойнты владельцу: карта подписи (пол) · мини-голд алиасов · precision@30 (books/gu-zhenren/exp16/). **Исследовательская программа ЗАВЕРШЕНА.** **Директива владельца (D39.11): стройка ТОЛЬКО после ДИЗАЙН-СИНТЕЗ сессии** (синтез всех находок → план реализации с верифицированными алгоритмами → ратификация → бэкенд-промт). **Очередь:** ✅ дизайн-синтез ИСПОЛНЕН, план РАТИФИЦИРОВАН (**D39.12** — `architecture/11-implementation-plan.md`, три рубежа верификации, 4 гейтнутых арма) → **бэкенд-пак ЕДИНОЙ сессией (промт ВЫДАН — `BACKEND_PLAN11_SESSION_PROMPT.md`; решение владельца)** → приёмка оркестратора → единый resnapshot (D30.9, одна переоплата; §8-манифест плана; вкл. RegressionGuard-тумблер D38.4) → пере-прогон 3–10 глав (армы редактора glm/mistral/deepseek-pro) → чтение владельца (интерим-планка 2 претензии, D35).残: POLYGON_PACKAGE4 residual-трекер (пилот/18+/echo — отложен, D36.1).
|
> - **exp15 ЗАВЕРШЁН И ЗАЛЕНДЕН (D39.7, `fe34a16`):** «граница вредит» отозван (артефакт окна судьи, свинг +0.689 после фикса), все когезия-эффекты под шум-полом 0.126, «sequential нигде не лучше» (LOO-stable, НЕ ратифицированный зелёный свет волне), фертильность 1.20·cjk+0.39·other; $12.79/$15. **Слепое чтение ИСПОЛНЕНО (D39.8):** планка не пройдена никем, дефекты распределены по армам (читатель подтвердил floor-null; V2-брак = риг без гейтов, не carryover-рычаг) → битва качества пере-прогона = дефект-классы (时辰-юниты · числа b1 · gender-enforce · стих/аллюзии · регистр-лексикон), не нарезка. **Q4a ИСПОЛНЕН И ЗАЛЕНДЕН (D39.9, $1.32/$2):** верность НЕ покупается на translate-стадии (flash уже верен 25/0/27; pro ×3.67 не ратифицирован); смысл ломает glm-РЕДАКТОР (свап glm→mistral/deepseek-pro = приоритетный арм пере-прогона); exp14b-правила дефектны → $0-ре-аудит в exp16. **exp16 ИСПОЛНЕН И ЗАЛЕНДЕН (D39.10, $0.02/$10):** WHICH подтверждена (код-детектор V-C recall 0.97 held-out, $0) / WHAT опровергнута для ко-оккуренции → dst-канал = банкнота (идея владельца оправдана) + облако+Палладий; 9B-споттер не нужен (0.895 ≤ код); Z1 → облако; ре-аудит exp14b: ядро D38 держится, errata не нужна. Тачпойнты владельцу: карта подписи (пол) · мини-голд алиасов · precision@30 (books/gu-zhenren/exp16/). **Исследовательская программа ЗАВЕРШЕНА.** **Директива владельца (D39.11): стройка ТОЛЬКО после ДИЗАЙН-СИНТЕЗ сессии** (синтез всех находок → план реализации с верифицированными алгоритмами → ратификация → бэкенд-промт). **Очередь:** ✅ план РАТИФИЦИРОВАН (D39.12) → ✅ **пак-11 Block A ПРИНЯТ и залендён (D39.13:** WS2 целиком [вкл. src→dst = закрытие D30.1] + WS1a/b/c-инфра + WS4-ядро; golden re-captured**)** → **продолжение-пак R1–R5** (волновой исполнитель · WS3-майнер · WS4-12-точек · WS5-чекеры · арм-конфиги; промт `BACKEND_PLAN11_SESSION_PROMPT.md` АКТИВЕН со статус-баннером + `backend/PACK11_REPORT.md` §Резидуал) → приёмка (с погашением долга рубежа-2) → единый resnapshot (D30.9, §8-манифест) → пере-прогон 3–10 глав (армы glm/mistral/deepseek-pro) → чтение владельца (интерим-планка 2 претензии, D35).残: POLYGON_PACKAGE4 residual-трекер (пилот/18+/echo — отложен, D36.1).
|
||||||
> - **Стек (не менялся с D38.5):** draft deepseek-v4-flash (thinking ON) → editor glm-5 БИЛИНГВ v3-discourse (P1a+чэнъюй+few_shot-тумблер; свап-кандидаты mistral/deepseek-pro = армы пере-прогона) → судья gemini-3.1-pro-preview; канал B Mistral+grok; 7 ключей; ~$0.85/ранобэ (D30.4).
|
> - **Стек (не менялся с D38.5):** draft deepseek-v4-flash (thinking ON) → editor glm-5 БИЛИНГВ v3-discourse (P1a+чэнъюй+few_shot-тумблер; свап-кандидаты mistral/deepseek-pro = армы пере-прогона) → судья gemini-3.1-pro-preview; канал B Mistral+grok; 7 ключей; ~$0.85/ранобэ (D30.4).
|
||||||
> - **Ждём от владельца:** запуск exp15 (бюджет ≤$15 подтверждён) · research/20 §E: мини-голд алиасов (~20–30 мин) / бюджет ≤$5 полигон-стадии / политика канона / жанр-паки / реплика ja→ru · чтение пере-прогона (после очереди) · старые висящие: планка запуска (лучше-фана/гибрид/издательский) · билингв-якорь пилота (D25.9-Q1) · контаминация пилот-корпуса (D27.4) · publishable/waiver (D25.1) · FN-bound L3 (D25.4) · юр-пакет · провенанс 12-*-доков.
|
> - **Ждём от владельца:** запуск exp15 (бюджет ≤$15 подтверждён) · research/20 §E: мини-голд алиасов (~20–30 мин) / бюджет ≤$5 полигон-стадии / политика канона / жанр-паки / реплика ja→ru · чтение пере-прогона (после очереди) · старые висящие: планка запуска (лучше-фана/гибрид/издательский) · билингв-якорь пилота (D25.9-Q1) · контаминация пилот-корпуса (D27.4) · publishable/waiver (D25.1) · FN-bound L3 (D25.4) · юр-пакет · провенанс 12-*-доков.
|
||||||
> - Архивы хроники: `archive/PROGRESS-2026-07-04-10.md` (D31) · `archive/PROGRESS-2026-07-10-13.md` (D39.6-гигиена). Записи ниже — живая эра D39.
|
> - Архивы хроники: `archive/PROGRESS-2026-07-04-10.md` (D31) · `archive/PROGRESS-2026-07-10-13.md` (D39.6-гигиена). Записи ниже — живая эра D39.
|
||||||
|
|
||||||
|
## Оркестратор №7 — пак-11 Block A принят и залендён (D39.13), 19.07 (третья фаза сессии)
|
||||||
|
|
||||||
|
Бэкенд-сессия сдала Block A + ядра с ЧЕСТНЫМ резидуалом (мандат «резидуал > сломанный суит»): WS2 целиком (output-бюджет чанкера, edit-единицы grouping-of-whole-chunks [девиация ратифицирована: 37 единиц / 0 straddle воспроизведены, гарантия W2-реконструкции по построению], src→dst-блок редактора = закрытие D30.1, `render-format-version`, golden re-capture) · WS1a eager-clients (гонка закрыта) · WS1b rate-guard (`ratelimit.go`) · WS1c base/enriched `MemoryVersion`-split (enriched байт-идентичен — снапшот не двинулся) · WS4-core банкнота-парсер (байт-верный порт). **Приёмка — верификация исполнением:** build/vet/race сам, 27 пинов свежим `-count=1` вербозно, независимая masked-golden репликация (все категории объяснимы), построчное чтение несущих диффов. Сессионный 9-агентный чекпойнт: 3 confirmed пофикшены / 3 refuted. **Долг: мульти-агентный рубеж-2 приёмки не гонялся (лимит-режим) — гасится при приёмке продолжения (прецедент D39.2).** Резидуал R1–R5 с точными дизайнами — `backend/PACK11_REPORT.md`; промт пака АКТИВЕН со статус-баннером. Записка продолжению-сессии: стартовать по промту + отчёту, сданное не пере-строить.
|
||||||
|
|
||||||
## Оркестратор №7 — план 11 РАТИФИЦИРОВАН (D39.12), бэкенд-промт единого пака ВЫДАН (19.07, вторая половина сессии)
|
## Оркестратор №7 — план 11 РАТИФИЦИРОВАН (D39.12), бэкенд-промт единого пака ВЫДАН (19.07, вторая половина сессии)
|
||||||
|
|
||||||
Дизайн-синтез сдал план (1066 строк) после трёх рубежей: ревью-2 оркестратора (F1–F11: 3 MAJOR соло-прохода + 2 замечания владельца [mistral rate-guard ~48% retry-fails · полы per-editor-model] + Q9-провенанс) → 7-агентный пост-фикс адверсариал сессии (5 находок: F8 ПЕРЕПИСАН — сайзинг уже от `prev`; дефолт 3200 → 37 edit-единиц; mined-write-путь [`loadGlossarySeed` хардкодит seed]; render-format-version реконсиляция; F6 усилен narrative-coref) → 3-агентный подтверждающий пас (2 резидуала закрыты). Спот-чек внесения — исполнением (36 маркеров, grep всех фиксов). **Залендено одним пакетом:** план с ревью-шапкой + `eval/design11/` (7 скриптов, 6 верификаций PASS) + quirks-запись mistral (из §12-заготовки, зона полигона — с пометкой) + D39.12 + архивация спент-промта дизайн-синтеза. **Гейтнутые армы: 4** (edit>3200 за Q2a · качество свап-арма за платным полигоном · DC-landing за $0-FP · mistral за rate-guard). Развилки §10 (14 шт., с рекомендациями) — у владельца, стройку не блокируют. **Выдан `BACKEND_PLAN11_SESSION_PROMPT.md`** — единый пак WS1–WS6 блоками A/B/C/D с чекпойнт-ревью (решение владельца: одной сессией; свободы: рефакторинг/перенос кода, временные тесты, багфиксы со скоуп-пометкой; Go-каноны; даблчек неконсистентностей; $0 к провайдерам). Очередь: бэкенд-сессия → приёмка (рубеж-2) → единый resnapshot → пере-прогон → чтение.
|
Дизайн-синтез сдал план (1066 строк) после трёх рубежей: ревью-2 оркестратора (F1–F11: 3 MAJOR соло-прохода + 2 замечания владельца [mistral rate-guard ~48% retry-fails · полы per-editor-model] + Q9-провенанс) → 7-агентный пост-фикс адверсариал сессии (5 находок: F8 ПЕРЕПИСАН — сайзинг уже от `prev`; дефолт 3200 → 37 edit-единиц; mined-write-путь [`loadGlossarySeed` хардкодит seed]; render-format-version реконсиляция; F6 усилен narrative-coref) → 3-агентный подтверждающий пас (2 резидуала закрыты). Спот-чек внесения — исполнением (36 маркеров, grep всех фиксов). **Залендено одним пакетом:** план с ревью-шапкой + `eval/design11/` (7 скриптов, 6 верификаций PASS) + quirks-запись mistral (из §12-заготовки, зона полигона — с пометкой) + D39.12 + архивация спент-промта дизайн-синтеза. **Гейтнутые армы: 4** (edit>3200 за Q2a · качество свап-арма за платным полигоном · DC-landing за $0-FP · mistral за rate-guard). Развилки §10 (14 шт., с рекомендациями) — у владельца, стройку не блокируют. **Выдан `BACKEND_PLAN11_SESSION_PROMPT.md`** — единый пак WS1–WS6 блоками A/B/C/D с чекпойнт-ревью (решение владельца: одной сессией; свободы: рефакторинг/перенос кода, временные тесты, багфиксы со скоуп-пометкой; Go-каноны; даблчек неконсистентностей; $0 к провайдерам). Очередь: бэкенд-сессия → приёмка (рубеж-2) → единый resnapshot → пере-прогон → чтение.
|
||||||
|
|
|
||||||
File diff suppressed because one or more lines are too long
Loading…
Add table
Reference in a new issue