Land exp15 REV.2: floor-gated corrected results with per-vote judge persistence, the retracted boundary headline, and orchestrator landing fixes to the deviations ledger
This commit is contained in:
Claude (backend session)
parent
cabdfb35b4
commit
fe34a16896
29 changed files with 5134 additions and 0 deletions
|
|
@ -311,3 +311,214 @@ within-chapter size-span ДО записи вывода. «0 ошибок ≠ 0
|
|||
8. **Судьи подтверждены живьём:** grok-4.3 reasoning-ON (262 rt), mistral-large-latest резолвится, glm-5 PAYG доступен. Gemini как 3-й судья — эмпир-верификация thoughts-accounting ДО опоры на его usage.
|
||||
9. **`reflow_omission.py` реюзабелен** для детерм. вторички (entity-omission), НО ПОТОЛОК (сущности выживают) — первичка Q1/Q3 = trap-точность span-судей; term-substitution (класс/разряд) = glossary-drift-ось, не omission.
|
||||
10. **Метрика границы Q1 (вторичка):** доля разрезов на scene-shift-швы (выше=лучше) vs continuation-швы (ниже=лучше); первичка — T-tense+T-ana точность (общий знаменатель в A0).
|
||||
|
||||
---
|
||||
|
||||
## §7 — ИСПОЛНЕНИЕ ПЛАТНОГО Q1/Q3 (полигон, 2026-07-18)
|
||||
|
||||
> Исполнено по хендофф-промту + курс-коррекциям оркестратора/владельца (три раунда решений по ходу).
|
||||
> Пре-рег §1 НЕ пере-сочинялся; все отклонения от него — явно в §7.7. Скрипты — `eval/exp15/` (ВНЕ git,
|
||||
> лендит оркестратор); артефакты/выходы — `/home/ubuntu/books/gu-zhenren/exp15/` (ВНЕ git).
|
||||
|
||||
> **⚠ РЕВИЗИЯ 2 (2026-07-18, пост-верификация оркестратора).** Оркестратор верифицировал первый прогон по
|
||||
> сырью (6 осей) и нашёл, что **три заголовка НЕ лендятся** — не из-за арифметики (она воспроизводится до
|
||||
> цента), а из-за **артефакта рига** и **пропущенного floor-гейта**. Ключевое: судейский head-excerpt
|
||||
> (`HEAD_CHARS=1100`) показывал судье лишь первые 1100 символов каждого финала; в оракул-арме зависимый
|
||||
> спан сидит в СЕРЕДИНЕ бо́льшего seam-выровненного чанка → его рендер уходил ЗА окно → судья видел
|
||||
> «отсутствует» = wrong/катастрофа ПО ПОСТРОЕНИЮ. Именно это, НЕ граница, дало заголовок «оракул −0.564».
|
||||
> Данная ревизия: **полно-чанковый пере-суд Q1b** (окно снято, весь финал), **полно-чанковый FLOOR** (A0
|
||||
> vs A0′, шум-пол), **re-analysis Q3a/Q0** (floor-relative, обе оценки, пул-fix-rate, TOST). Первый прогон
|
||||
> (§7.4-v1) сохранён как провенанс. Полный список правок — §7.8. Пре-рег §1 НЕ пере-сочинялся.
|
||||
|
||||
### §7.0 Итог одной строкой (РЕВ.2)
|
||||
Судейский **шум-пол ≈ 0.126** (FLOOR: A0 vs A0′, mean|Δ|; сходится с 0.119/0.115 из двух независимых прокси)
|
||||
**больше любого маргинального эффекта** на этом срезе → **ни граница (Q1), ни carryover (Q3a), ни редактор
|
||||
(Q0) не отличимы от судейского шума**. Заголовок v1 «оракул хуже (−0.564)» = **артефакт head-excerpt** (пере-суд
|
||||
полными чанками сдвинул на **+0.689** → T-ana **+0.125** на самом полу, LOO-хрупко): вывод «граница ВРЕДИТ»
|
||||
**отозван** — seam-выравнивание НЕ вредит и НЕ является ратифицируемым рычагом; единственный устойчивый сигнал —
|
||||
слепые greedy-разрезы иногда **рвут сущностную кореференцию** (T-ent катастрофы flat 15 vs оракул 1). Базовый
|
||||
flat-A0 (greedy-1500 + глоссарий) держит бо́льшую часть кросс-граничных анафор — кросс-граничные сбои РЕДКИ.
|
||||
Ось решения — **слепое чтение владельца** (метрики номинируют, чтение ратифицирует) + больший/трудный корпус.
|
||||
|
||||
### §7.1 Материал (актуальный — отклонение от пре-рег S2′-12节)
|
||||
- Пре-рег S2′ (12 节, склейка) нёс **пустые строки на швах** (`\n\n\n`) → бэкендовый `splitParagraphs` видит
|
||||
их как абзацы → greedy A0 режет на ВСЕХ 11 швах → **премиса Q1 нарушена** (A0 не «слеп» к сценам). Диагностика
|
||||
байт-верифицирована (Δ2=индент на всех швах; §7.7-D2).
|
||||
- **Фикс (решение оркестратора):** collapse `\n\n→\n` = **flat** (швы неявные, greedy слеп). Старый blank-line
|
||||
анкор (aligned_a0, режет на истинных швах) **не выброшен — стал ОРАКУЛ-армом Q1b** (оракул-коррекция).
|
||||
- **Расширение (решение владельца, полная мощность):** срез **节 1–45→30** (после проекции time/money).
|
||||
Итог: **S2prime_big_flat.txt** (30 节, 87.5k рун, **flat-A0-30 = 51 чанк** greedy-1500) + **blanklines**-версия
|
||||
(**oracle-30 = 63 чанка**, режет на 29 швах + бюджет). Sid-покрытие здорово до 节60 (медиана ~16 термов/节) *[испр. оркестратором: расчёт не персистирован — сессионный; к пере-выводу скриптом при переиспользовании]*.
|
||||
|
||||
### §7.2 Порт инъекции — валидирован БАЙТ-В-БАЙТ (несущее)
|
||||
`eval/exp15/mem_select.py` — полный Python-порт Go `Select` (memnorm нормализация + Aho-Corasick +
|
||||
trust-gated `suppressContained` + sticky depth-2 + budget + render). Валидация **методом оркестратора**
|
||||
(`LOG_LLM_BODIES=1 LOG_LEVEL=debug LOG_FORMAT=json` → байт-дифф 2-го system-msg против отрендеренного Go-блока):
|
||||
**L1 injected_ids + L2 translator-блок = 24/24 (оракул), 51/51 (flat-A0-30), 51/51 (flat-A0′-30)** — БАЙТ-идентично.
|
||||
Ключи к идентичности: frozen entry order = `ORDER BY src,sense,since_ch,until_ch,status,dst` (glossary.go:178);
|
||||
`injected_ids` слеп к sticky (валидировать против блока, НЕ ids). Greedy-чанкер тоже портирован
|
||||
(`chunker.py greedy_chunks`, воспроизводит A0-границы точно). Арм-риг промпт байт-совпал с бэкендом (773c translator).
|
||||
|
||||
### §7.3 Трап-сет (детерм-первый + kimi-обогащение; guard 1/3)
|
||||
`final_traps.json` — **44 трапа, все straddling flat-A0-30-границу** (общий знаменатель):
|
||||
- **T-ana = 19** (первичка, carryover-дискриминирующая): 7 детерминистических (anaphor-head) + 12 kimi-обогащения
|
||||
на границах (батчами, $0.68, 0 ретраев). *19 — на 1 ниже квоты ≥20; честно-недомощно.*
|
||||
- **T-ent = 23** (NEGATIVE CONTROL): глоссарий-инъекция уже держит каноническую dst в обоих чанках → carryover
|
||||
не должен добавить; контроль квантует size-конфаунд.
|
||||
- **T-ell = 2** (вторичка; коротко — honest-power).
|
||||
- **T-tense/T-cat/T-gen/T-reg = «undecidable a priori на материале»** (guard 3): плотность 3/80 в майнинге =
|
||||
свойство прозы, не размер среза; НЕ имитировали.
|
||||
- Найдено ключевое: kimi-майнинг (11 окон, $1.45) дал 80 кандидатов, но ЛОКАЛЬНЫХ (не straddling) — 4/77
|
||||
пересекали границу; детерминистический entity/anaphor-детект на границах + таргет-обогащение — верный метод.
|
||||
|
||||
### §7.4 Результаты по Q (РЕВ.2 — floor-relative; судьи grok-4.3 reasoning-ON + mistral-large-latest, оба порядка, 6→10, LOO, катастроф-скрин)
|
||||
|
||||
**§7.4.0 FLOOR — судейский шум-пол (A0 vs A0′, полные чанки, $1.24, 404 вызова, n=21 первичных, 0 транспорт-ошибок).**
|
||||
A0 и A0′ = два независимых прогона ОДНОГО конфига (разный семпл-сид) → истинно эквивалентны; любой |Δ| = шум
|
||||
судьи+генерации. Результат: **mean|Δ| = 0.126** (median 0.000, max 1.0 — бимодально: 12/21 ячеек ровно 0, хвост
|
||||
до полного разворота), signed mean **−0.019** (нет системного смещения), катастрофы **58/58 симметрично**. Сходится
|
||||
с прокси a0↔a0 из Q3a-данных (0.115) и оценкой оркестратора (0.119). **ГЕЙТ ИНТЕРПРЕТАЦИИ (фриз §1.5): любой
|
||||
|эффект| < ~0.126 не отличим от судейского шума.**
|
||||
|
||||
**Q1a — детекция ($0):** §B1.5-DP-чанкер восстанавливает **0/7** истинных швов на flat (точное рунное совпадение;
|
||||
= слепой greedy, тоже 0/7). *[v1 сообщал 1/7 — из СТЕЙЛ `q1_setup.json` (mtime до flat-фикса); пере-прогон рига
|
||||
даёт 0/7 у обоих.]* Судей не жгли.
|
||||
|
||||
**Q1b — граница (оракул seam-aligned vs flat blind). ⚠ ПЕРЕ-СУЖДЕНО ПОЛНЫМИ ЧАНКАМИ ($1.72, 556 вызовов, 0
|
||||
транспорт-ошибок).** [+diff = оракул лучше; сравнение с 0 + шум-полом 0.126]
|
||||
| тип | n | mean_diff (v2 полн-чанк) | CI90 | катастрофы оракул/flat | v1 (head-excerpt, АРТЕФАКТ) |
|
||||
|---|---|---|---|---|---|
|
||||
| **T-ana (первичка)** | 18 | **+0.125** | [+0.008, +0.243] | 55/71 | ~~−0.564~~ |
|
||||
| T-ana+T-ell (primary) | 20 | **+0.072** | [−0.052, +0.196] | 65/72 | — |
|
||||
| T-ent (контроль) | 10 | +0.112 | [−0.001, +0.225] | **1/15** | ~~−0.308~~ |
|
||||
| T-ell | 2 | ≈−0.4 | (n=2) | — | ~~−0.834~~ |
|
||||
|
||||
- **Артефакт head-excerpt подтверждён и снят.** v1 заголовок T-ana **−0.564** был целиком следствием усечения
|
||||
1100 символов (в оракуле зависимый спан — в середине, за окном → судья «не видит» → wrong/катастрофа). Полный
|
||||
чанк сдвинул T-ana на **+0.689** (полная смена знака). Диагностический тест: split по позиции зависимого в
|
||||
оракул-чанке — v1 дал **dep-в-окне +0.238 / dep-вне −0.872** (сигнатура артефакта); v2 полными чанками
|
||||
**+0.176 / +0.028** — split **СХЛОПНУЛСЯ** (артефакт устранён). Катастроф-асимметрия v1 (168/43) тоже была
|
||||
артефактом → v2 почти симметрично (65/72).
|
||||
- **Скорректированный вывод: seam-выравнивание НЕ вредит и НЕ является ратифицируемым рычагом.** Первичка
|
||||
(T-ana+T-ell) **+0.072 — В ПРЕДЕЛАХ шум-пола 0.126**; T-ana **+0.125 на самом полу** и **LOO-хрупко**: grok видит
|
||||
оракул лучше (+0.307), mistral ~нейтрально (−0.029) — судьи расходятся по знаку. T-ell (n=2) оба судьи ниже нуля.
|
||||
- **Единственный устойчивый сигнал** (оба судьи согласны, +0.11): на **T-ent** слепые greedy-разрезы иногда рвут
|
||||
сущностную кореференцию через границу → катастрофы (T-ent-1 5, T-ent-6 1, T-ent-12 9 = flat 15), которых
|
||||
seam-выравнивание избегает (оракул 1). Это реальный механизм, но класс — негативный контроль, эффект на полу.
|
||||
**НЕ вычитаем T-ent из T-ana** (пост-хок size-контроль −0.31 из v1 сам нёс тот же артефакт — не лендится).
|
||||
- Оговорка (оркестратор): **17/18 трапов оракул НЕ разрезал на швах** (0 на seam) → механизм «оракул режет
|
||||
continuation-клиффхэнгеры» НЕ протестирован ни одним трапом; тест — про качество когезии ВНУТРИ разного чанкинга.
|
||||
T-ana-33 исключён из Q1b (оракул-чанк 12 = cjk_artifact, editor-skip; оракул-чанк 42 — тот же класс, трапов на него не мапилось). *[испр. оркестратором:]* T-ana-33 присутствует ТОЛЬКО в FLOOR; Q3a/Q0/Q1b-v1 судились на n=18 (в сессии-1 он не входил ни в один блок).
|
||||
|
||||
**Q3a — carryover (пере-АНАЛИЗ существующих данных; артефакт head-excerpt Q1b-специфичен — у Q3a обе стороны
|
||||
flat-чанкинг, зависимый в ГОЛОВЕ у обеих → усечения-асимметрии нет; $2.32, 1303 голоса).** Обе оценки + CI90:
|
||||
- **Первичка (T-ana+T-ell), парная**: src **−0.005** [−0.12,+0.11], draft −0.008, final **−0.055**; маргинальная
|
||||
arm-acc ≈ парной (src −0.005). **ВСЕ ниже шум-пола 0.126 → не интерпретируемы.** (Заголовок v1 «+0.042» —
|
||||
пост-хок ТОЛЬКО T-ana; по пре-рег §1.5 первичка = T-ana+T-ell.)
|
||||
- **Преимущества последовательного final-соседа НЕ обнаружено НИГДЕ**: точечная оценка ≤0 в обеих метриках и во
|
||||
всех LOO-срезах (final: парная −0.055, LOO без-grok −0.061 / без-mistral −0.038). «Цена параллелизма ≈0» v1 —
|
||||
подмена метрики (arm-acc −0.0002 vs парная −0.047 для final); честно: **цена параллелизма НЕ исключена до
|
||||
~0.05–0.10** (MDE≈0.18, p≈0.5); «не ратифицированный зелёный свет», не «безопасно».
|
||||
- **§D5 fix-rate = пул-определение 2/6 = 33% во ВСЕХ режимах** (v1 «25/25/0%» — артефакт пере-суженных per-mode
|
||||
знаменателей: A0 пере-суждён отдельно на режим). A0-провалено 6 трапов (пул a0<0.5).
|
||||
- **Раскрытие T-ell-43** (катастрофа «победителя»): a0 1.0 → арм src 0.0 / draft 0.125 / final 0.042; src
|
||||
cat_arm 11 vs a0 0 — source-carryover САМ создал грубую ошибку там, где flat был идеален.
|
||||
- LOO src меняет знак по судье; final устойчиво-отрицателен.
|
||||
|
||||
**Q0 — do-nothing (draft) vs A0 (editor final); эквивалентность НЕ установлена ($0.99, 578 голосов).** Точка
|
||||
mean(A0−do-nothing) **+0.041** верна, но **TOST ПРОВАЛЕН**: sd 0.155, **90% CI (−0.008, +0.089)** — верх за
|
||||
маржой M=0.05 → по правилу фриза **«данных мало», ветка §D5 «мандаты редактора пересматриваются» НЕ триггерится**.
|
||||
Симметрия (в заголовок рядом с carryover-плюсом): редактор **режет катастроф-голоса** T-ana 66→52 (−21%, размер
|
||||
как у Q3a-плюса). Единицы: это ГОЛОСА, не трапы (distinct трапов-с-катастрофой editor 8 vs donothing 11); редактор
|
||||
добавил флаги на 5/18 трапов, чистый срез тянут T-ana-32 (13→0) и T-ell-40 (6→0). ⚠ Каветка v1 в силе: трап-сет —
|
||||
когезия, которую редактор и не адресует (стиль/reflow/глоссарий-энфорс ЗДЕСЬ не мерен).
|
||||
|
||||
### §7.5 Дерево решений §D5 (РЕВ.2 — floor-relative)
|
||||
- **Q1 (граница):** первичка **+0.072 в пределах шум-пола** + Q1a 0/7 → **строить §B1.4/B1.5-DP НЕ показано; граница
|
||||
не является ратифицируемым рычагом когезии на этом срезе.** Но вывод v1 «граница ВРЕДИТ» **ОТОЗВАН** (был артефакт):
|
||||
seam-выравнивание нейтрально-к-слабо-позитивно и режет сущностные катастрофы разрезов. Практический итог тот же
|
||||
(граница вторична), но по ВЕРНОЙ причине (sub-floor null, не вред).
|
||||
- **Q3a (carryover):** все эффекты ниже пола; преимущества последовательного final-соседа не обнаружено нигде
|
||||
→ **оживление carryover НЕ ратифицируется**; source-carryover в волне 1: вреда не видно выше шум-пола (MDE≈0.18 — вред до ~0.05–0.10 не исключён), пользы не доказано. Недомощно
|
||||
(база держит бо́льшую часть, ~6 провалов) → человеческий эндпоинт + больший/трудный корпус.
|
||||
- **Q0 (редактор):** эквивалентность на когезия-трапах **НЕ установлена** (CI за M) → **мандаты редактора НЕ
|
||||
пересматриваются по этим данным** (данных мало); редактор режет катастроф-хвост; его стиль/reflow-ценность — ВНЕ
|
||||
этого трап-сета (не мерена).
|
||||
- **Финал:** ни одна ветка не даёт прод-кандидата ТОЛЬКО по метрикам (все — на/под судейским полом); гейтящий
|
||||
эндпоинт — **слепое чтение владельца** (`exp15/monitor/blind_read_packet.md`, метки перемешаны; ключ отдельно) +
|
||||
больший корпус. Метрики НОМИНИРУЮТ, чтение РАТИФИЦИРУЕТ (планка ≤2 претензии/глава).
|
||||
|
||||
### §7.6 Деньги (РЕВ.2)
|
||||
Глобальный спент **$12.79 / $15** (маржа ~$2.2). Сессия-1 (первый прогон) $9.82 [авторитетный учёт оркестратора,
|
||||
2541 биллинг-строка, 0 расхождений — включает tmctl-генерацию анкоров, которой НЕТ в Python-леджерах]. Сессия-2
|
||||
(ревизия): **FLOOR полн-чанк $1.24 + Q1b пере-суд полн-чанк $1.73 = $2.97**. Гейты: пер-call predicted-cost
|
||||
($0.04) + **ГЛОБАЛЬНЫЙ потолок $14.5** (seed = $9.82 + новые леджеры; фикс review-I8: раньше per-block потолок
|
||||
суммировался в ~4×бюджета). Ошибки: **0 транспорт-ошибок; 3 parse-drop / 2313 голосов = 99.87% успех** (v1 «0
|
||||
ошибок» уточнён). **«~46 таймаутов» УБРАН** — артефакта нет (max latency 66.5s при timeout 240s; литеральных
|
||||
таймаутов 0). Rate-limit: FLOOR полн-чанк показал mistral 48% retry-fails @1.3s (max lat 64.7s), grok 0% →
|
||||
интервал mistral расширен до 2.6s для Q1b *[испр. оркестратором: retry-fail-rate не изменился, 49.5%→48.2% — эффект интервала не подтверждён леджерами; транспорт-ошибок 0, все ретраи успешны]*. **Q4a вынесен ВНЕ этого капа** (отдельная долинная
|
||||
мини-сессия, ~$2, дефолт-да владельца — §7.8/дев.9).
|
||||
|
||||
### §7.7 Самопроверка + ОТКЛОНЕНИЯ от пре-рега (явно)
|
||||
**Ревью ИСПОЛНЕНИЕМ поймал 9 багов ДО искажения платных результатов** (мандат самопроверки в действии):
|
||||
адверсариальный ревью рригов (5) + sanity-гейты (4). Ключевые:
|
||||
- (A) **arms.py editor-context double-inject** (HIGH) — 3 q3a-режима слали контекст в оба стейджа; фикс: shared
|
||||
src-carryover draft → 3 редактора (только target-side разнится). Поймано sanity-гейтом.
|
||||
- (B) **predict_cost под-предсказывал kimi-reasoning** → потом ЭТОТ фикс уронил grok в судьях (гейт-блок $0.033>$0.03);
|
||||
фикс: judge max_tokens=1500. Поймано judge-selftest'ом.
|
||||
- (C) mistral 429 rate-limit → retry+backoff+per-model delay. (D) trap_curate `cut<=hi` off-by-one + grid-dedup.
|
||||
(E) arms draft finish=length → draft_max 16000. (F) mem_select entry-order (byte-diff). (G) mem_select
|
||||
editor transitive-валидация. (H) S2′ CRLF/seam-offset дрейф (36186 vs 35289). (I) стейл seam-файл.
|
||||
**Отклонения от §D/пре-рег (санкционированы оркестратором/владельцем):**
|
||||
1. S2′-12节 → **flat 30节** (материал-баг + полная мощность); aligned = оракул-арм (не выброшен).
|
||||
2. Механизм-армы = eval-риги (бэкенд не умеет), НО инъекция БАЙТ-валидирована против Go (не «верить на слово»).
|
||||
3. Трапы: **детерм-первый** (entity-straddle + anaphor-head) + kimi таргет-обогащение НА границах (не seam-окна).
|
||||
4. **T-tense/cat/gen/reg = undecidable a priori** (плотность прозы, не размер) — НЕ имитированы.
|
||||
5. Q1 декомпозирован: Q1a-детекция ($0) + Q1b-оракул-контраст. Судьи grok+mistral (апекс-gemini инфизибл, §1.12).
|
||||
6. T-ana=19 (на 1 ниже квоты). Голоса 6→10, оба порядка, LOO — как §1.6.
|
||||
7. **Q4a вынесен ВНЕ капа exp15** в отдельную долинную мини-сессию (~$2, дефолт-да владельца; пин: draft =
|
||||
deepseek-v4-pro, суб-арм do-nothing обязателен, судейские окна полно-evidence — HEAD_CHARS-класс НЕ наследовать).
|
||||
Дизайн — Q4a-дополнение оркестратора (концерн №1 «сильный переводчик»; слой-3 контракт t/e). Причина выноса:
|
||||
полн-чанк Q1b+FLOOR при капе $15 не оставляют Q4a места без урезаний — решение оркестратора/владельца (не резать).
|
||||
8. Q2a/Q2c/Q3b/Q3a′/Q5 — не исполнены (бюджет/приоритет; §D2-порядок, стоп-гейт легитимен).
|
||||
9. **T-ent-контроль кап 10/23** (только код `--tent-control`; для бюджета судейского сэмпла) — раскрыто.
|
||||
10. **Маркер §1.14г:** материал S2′ — СИНТЕТИЧЕСКАЯ сцен-склейка + PD-претрейн-канон → все эмпирические выводы §7
|
||||
**ПРЕДВАРИТЕЛЬНЫ** до вебновелл-среза (не претрейн). Планка чистовика — тот же пилот на непретрейн-корпусе.
|
||||
11. **D30.10 (леджер-дисциплина):** ad-hoc kimi-пробы (~$0.24) шли без леджера — впредь персистить; Q1b-v1-леджер
|
||||
сохранён как `judge_costs_Q1b_v1_headexcerpt.jsonl` (НЕ сброшен — провенанс артефакта); v2 пишет свежий.
|
||||
|
||||
#
|
||||
*[испр. оркестратором при лендинге — дополнение списка девиаций по пере-верификации:]*
|
||||
12. **Q1b-пере-суд отказался от пре-рег size-matched ~3–4k-бюджета** (§1.5/§1.14в): судились greedy-1500 flat (51 чанк) vs oracle (63); честность размера держится на парности судимых чанков (flat≈oracle 0.95×), не на пре-рег дизайне.
|
||||
13. **Holm-коррекция (§1-пин) нигде не применялась** — репортились CI90 без множественной коррекции.
|
||||
14. Восстановление T-ell в primary и пропуск floor-гейта в v1 — задокументированы в §7.8, входят в этот список по ссылке.
|
||||
|
||||
## §7.8 Раунд коррекции методологии (2-й рубеж самопроверки: верификация оркестратора → фикс-лист)
|
||||
Мандат самопроверки сработал на ДВУХ рубежах: (1) сессия-1 — ревью исполнением поймал 9 багов ДО спенда (§7.7);
|
||||
(2) **пост-хок верификация оркестратора при лендинге (author≠reviewer)** — 6 осей по сырью: арифметика чистая
|
||||
(2541 строка, 0 расхождений), судейская механика чистая (оба порядка 69/70, эскалация-инварианты 0 нарушений),
|
||||
но **3 заголовка не лендятся**. Это ровно тот случай, ради которого правило существует.
|
||||
|
||||
**Что чинилось (полн-чанк пере-прогон, санкция оркестратора; пре-рег НЕ пере-сочинялся):**
|
||||
- **Q1b CRITICAL:** `HEAD_CHARS=1100` усекал финал → в оракуле зависимый спан за окном → −0.564 = артефакт.
|
||||
Фикс: **полные чанки** (окно снято; flat≈oracle по размеру 5565/5263 → size-конфаунда нет; зависимый+якорь
|
||||
всегда видны). Пере-суд: T-ana −0.564 → **+0.125** (сдвиг +0.689); dep-split схлопнулся. Заголовок отозван.
|
||||
- **Пропущенный floor-гейт:** добавлен **FLOOR-блок** (A0 vs A0′ полн-чанк) → шум-пол 0.126 → все маргиналы
|
||||
Q3a/Q0 sub-floor (не интерпретируемы). Это пре-регнутый §1.5-гейт, которого в v1 не было.
|
||||
- **Q3a/Q0 пере-анализ** (без пере-суда — артефакт Q1b-специфичен): первичка = T-ana+T-ell (не пост-хок T-ana),
|
||||
обе оценки+CI, пул-fix-rate 2/6, T-ell-43 раскрыт, TOST (Q0 экв. НЕ установлена).
|
||||
|
||||
**Адверсариальный ревью риггов (2-й, ДО спенда, 4 линзы + синтез)** — 10 находок; реальные закрыты $0-верифицированно:
|
||||
- **windowing** (I3): oracle-выбор чанка не проверял якорь (короткие пробы <8 симв. коллидируют) → **anchor-guard**
|
||||
(чанк с ЯКОРЕМ+зависимым; иначе skip). Дал повод перейти на полные чанки (I7: FLOOR не валидировал бы окна).
|
||||
- **per-vote** (I4): cat булев через bool() («"false"»→True) + tolerant-parse ронял cat/ev → нормализация +
|
||||
regex-восстановление. Плюс **per-vote персист** (сырые вердикты в результатах — катастроф-LOO аудит).
|
||||
- **budget** (I8): per-block потолок суммировался в ~4×бюджета → **глобальный seed** ($9.82 floor + новые леджеры,
|
||||
потолок $14.5); gate-block отделён от parse-fail (`budget_truncated`-флаг — не скорить недобранную ячейку);
|
||||
retry-fails в леджер (видимость).
|
||||
- **rate-limit** (эмпирика FLOOR): mistral 48% retry @1.3s → интервал 2.6s.
|
||||
|
||||
Все скрипты пере-прогона: `eval/exp15/{judge_run,judges,exp15_llm,q1b_reanalysis,q3a_reanalysis,q0_reanalysis}.py`.
|
||||
Артефакты: `judge_results_{FLOOR,Q1b}.json`, `q{1b,3a,0}_reanalysis.json`, `S2prime_big_seam_profile_1_29.json`
|
||||
(швы 12–29 доклассифицированы: 30节 = 15 scene-shift / 14 continuation / soft {1,7,15}), v1-провенанс
|
||||
`*_v1_headexcerpt.*`. Слепой пакет владельца НЕ трогался (его эндпоинт — полные тексты, чист).
|
||||
|
|
|
|||
181
eval/exp15/anchors_analyze.py
Normal file
181
eval/exp15/anchors_analyze.py
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — extract A0/A0' from the real backend DBs+logs and compute:
|
||||
- per-chunk SOURCE (recovered from draft user-msg heads), DRAFT, FINAL (editor) texts
|
||||
- greedy cut offsets (A0 boundaries — the Q1 common-denominator boundaries) + est_out distribution
|
||||
- deterministic KPI per whole-anchor (mean_spp, 1-sent-para frac, dialogue-dash frac, CJK-leak,
|
||||
glossary/term consistency vs seed)
|
||||
- A0<->A0' deterministic divergence (chunk count, boundary offsets, KPI deltas, term consistency,
|
||||
char-level similarity) = PRELIMINARY noise floor (the judged trap floor is measured with the judges)
|
||||
|
||||
Chunk SOURCE recovery reuses validate_injection.recover_chunk_sources (draft user-msg head -> offset in
|
||||
S2'.txt). FINAL text = the edit-stage response_text per chunk (the authoritative editor output).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from validate_injection import load_exchanges, recover_chunk_sources
|
||||
from chunker import est_out, est_tokens
|
||||
import regex
|
||||
|
||||
S2 = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime.txt")
|
||||
_RE_HAN = regex.compile(r"\p{Han}")
|
||||
_RE_CYR = regex.compile(r"[А-Яа-яЁё]")
|
||||
|
||||
|
||||
def final_texts(db):
|
||||
"""chunk_idx -> editor (final) response_text, latest attempt. S2' is one chapter."""
|
||||
con = sqlite3.connect(db)
|
||||
rows = con.execute(
|
||||
"SELECT chunk_idx, attempt, response_text FROM checkpoints WHERE stage='edit' ORDER BY chunk_idx, attempt"
|
||||
).fetchall()
|
||||
con.close()
|
||||
out = {}
|
||||
for ck, att, txt in rows:
|
||||
out[ck] = txt # later attempt overwrites
|
||||
return out
|
||||
|
||||
|
||||
def draft_texts(db):
|
||||
con = sqlite3.connect(db)
|
||||
rows = con.execute(
|
||||
"SELECT chunk_idx, attempt, response_text FROM checkpoints WHERE stage='draft' ORDER BY chunk_idx, attempt"
|
||||
).fetchall()
|
||||
con.close()
|
||||
out = {}
|
||||
for ck, att, txt in rows:
|
||||
out[ck] = txt
|
||||
return out
|
||||
|
||||
|
||||
def load_anchor(db, log):
|
||||
src_text = S2.read_text(encoding="utf-8")
|
||||
exchanges = load_exchanges(Path(log))
|
||||
srcs = recover_chunk_sources(exchanges, src_text, whole_file=False) # {(chap,ck): source}
|
||||
finals = final_texts(db)
|
||||
drafts = draft_texts(db)
|
||||
chunks = []
|
||||
for (chap, ck) in sorted(srcs.keys()):
|
||||
src = srcs[(chap, ck)]
|
||||
chunks.append({"chapter": chap, "chunk": ck, "source": src,
|
||||
"draft": drafts.get(ck), "final": finals.get(ck),
|
||||
"src_est_out": est_out(src) if src else None,
|
||||
"src_est_tokens": est_tokens(src) if src else None})
|
||||
return chunks, src_text
|
||||
|
||||
|
||||
def cut_offsets(chunks):
|
||||
"""Rune offsets in S2' where each greedy cut falls (chunk starts, excluding 0)."""
|
||||
offs, cur = [], 0
|
||||
for c in chunks:
|
||||
if c["source"] is None:
|
||||
return None
|
||||
if cur > 0:
|
||||
offs.append(cur)
|
||||
cur += len(c["source"])
|
||||
return offs
|
||||
|
||||
|
||||
# --- deterministic KPI ------------------------------------------------------------------------------
|
||||
def paragraphs(text):
|
||||
return [p for p in re.split(r"\n\s*\n|\n", text) if p.strip()]
|
||||
|
||||
|
||||
def sentences_ru(text):
|
||||
return [s for s in re.split(r"(?<=[.!?…])\s+", text) if s.strip()]
|
||||
|
||||
|
||||
def kpi(final_text):
|
||||
paras = paragraphs(final_text)
|
||||
spp = [len(sentences_ru(p)) for p in paras] or [0]
|
||||
mean_spp = sum(spp) / len(spp)
|
||||
one_sent = sum(1 for x in spp if x == 1) / len(spp) if spp else 0
|
||||
dash_paras = sum(1 for p in paras if p.lstrip().startswith(("—", "–", "-")))
|
||||
dash_frac = dash_paras / len(paras) if paras else 0
|
||||
# CJK leak: Han runes in the RU output NOT immediately after a Cyrillic word in parens gloss
|
||||
# (owner decision #3: «Кулак Ло Ханя (羅漢拳)» balanced Han-in-parens is NOT a leak). Crude: count
|
||||
# Han runes that are inside parentheses -> whitelisted; Han outside parens -> leak.
|
||||
han_total = len(_RE_HAN.findall(final_text))
|
||||
han_in_parens = 0
|
||||
for m in re.finditer(r"[（(]([^）)]*)[）)]", final_text):
|
||||
han_in_parens += len(_RE_HAN.findall(m.group(1)))
|
||||
han_leak = han_total - han_in_parens
|
||||
return {"n_paras": len(paras), "mean_spp": round(mean_spp, 2), "one_sent_para_frac": round(one_sent, 3),
|
||||
"dialogue_dash_frac": round(dash_frac, 3), "han_total": han_total,
|
||||
"han_in_parens": han_in_parens, "han_leak_outside_parens": han_leak,
|
||||
"ru_chars": len(_RE_CYR.findall(final_text))}
|
||||
|
||||
|
||||
def term_consistency(final_text, seed_path):
|
||||
"""LTCR-style: for each approved seed term present in S2', is its canonical dst (or a decl form)
|
||||
present in the final? Reports coverage and any missing (a drift/omission signal, deterministic)."""
|
||||
import yaml
|
||||
d = yaml.safe_load(Path(seed_path).read_text(encoding="utf-8"))
|
||||
src_text = S2.read_text(encoding="utf-8")
|
||||
hits, miss = [], []
|
||||
for t in d["terms"]:
|
||||
src = t.get("src"); dst = (t.get("dst") or "").strip()
|
||||
if not src or not dst or (t.get("status") or "approved") != "approved":
|
||||
continue
|
||||
variants = [src] + [a.get("alias") for a in (t.get("aliases") or []) if a.get("alias")]
|
||||
if not any(v in src_text for v in variants):
|
||||
continue
|
||||
forms = set([dst] + list((t.get("decl") or {}).get("forms") or []))
|
||||
# match the distinctive first content token of the dst (case-insensitive)
|
||||
present = any(f.lower() in final_text.lower() for f in forms)
|
||||
if not present:
|
||||
# fallback: distinctive stem (first word >=4 chars)
|
||||
toks = [w for w in re.findall(r"[А-Яа-яЁё]+", dst) if len(w) >= 4]
|
||||
present = any(w.lower()[:5] in final_text.lower() for w in toks)
|
||||
(hits if present else miss).append(dst)
|
||||
return {"n_terms_in_source": len(hits) + len(miss), "present": len(hits),
|
||||
"missing": miss, "coverage": round(len(hits) / (len(hits) + len(miss)), 3) if (hits or miss) else None}
|
||||
|
||||
|
||||
def main():
|
||||
seed = "/home/ubuntu/books/gu-zhenren/guzhenren-seed-v2.yaml"
|
||||
AH = "/home/ubuntu/books/gu-zhenren/exp15/anchors"
|
||||
out = {}
|
||||
for tag in ("flat_a0", "flat_a0prime"):
|
||||
chunks, _ = load_anchor(f"{AH}/{tag}.db", f"{AH}/{tag}.log.json")
|
||||
whole_final = "\n\n".join(c["final"] for c in chunks if c["final"])
|
||||
offs = cut_offsets(chunks)
|
||||
src_outs = [c["src_est_out"] for c in chunks if c["src_est_out"]]
|
||||
out[tag] = {
|
||||
"n_chunks": len(chunks),
|
||||
"cut_offsets": offs,
|
||||
"src_est_out": {"mean": round(sum(src_outs) / len(src_outs), 1), "min": round(min(src_outs), 1),
|
||||
"max": round(max(src_outs), 1), "per_chunk": [round(x, 1) for x in src_outs]},
|
||||
"kpi": kpi(whole_final),
|
||||
"term_consistency": term_consistency(whole_final, seed),
|
||||
"final_chars": len(whole_final),
|
||||
}
|
||||
Path(f"{AH}/{tag}.final.txt").write_text(whole_final, encoding="utf-8")
|
||||
Path(f"{AH}/{tag}.chunks.json").write_text(
|
||||
json.dumps([{k: c[k] for k in ("chunk", "source", "draft", "final", "src_est_out")} for c in chunks],
|
||||
ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
# A0<->A0' deterministic floor
|
||||
a, b = out["flat_a0"], out["flat_a0prime"]
|
||||
floor = {
|
||||
"same_chunk_count": a["n_chunks"] == b["n_chunks"],
|
||||
"same_cut_offsets": a["cut_offsets"] == b["cut_offsets"],
|
||||
"n_chunks": [a["n_chunks"], b["n_chunks"]],
|
||||
"kpi_delta": {k: round(a["kpi"][k] - b["kpi"][k], 3) for k in ("mean_spp", "one_sent_para_frac",
|
||||
"dialogue_dash_frac", "han_leak_outside_parens") if isinstance(a["kpi"][k], (int, float))},
|
||||
"term_coverage": [a["term_consistency"]["coverage"], b["term_consistency"]["coverage"]],
|
||||
"final_chars": [a["final_chars"], b["final_chars"]],
|
||||
}
|
||||
result = {"anchors": out, "deterministic_floor_A0_vs_A0prime": floor}
|
||||
Path(f"{AH}/anchors_analysis.json").write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(json.dumps({"a0_chunks": a["n_chunks"], "a0prime_chunks": b["n_chunks"],
|
||||
"a0_cut_offsets": a["cut_offsets"], "same_boundaries": floor["same_cut_offsets"],
|
||||
"a0_src_est_out_mean": a["src_est_out"]["mean"],
|
||||
"floor": floor}, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
265
eval/exp15/arms.py
Normal file
265
eval/exp15/arms.py
Normal file
|
|
@ -0,0 +1,265 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — Q3 cohesion arm-runner (eval-rig; design §B3/§D2). The backend can't do carryover/lookahead
|
||||
(dead knobs), so Q3 arms are eval-rigs that reuse A0's EXACT prompts + validated mem_select injection,
|
||||
adding ONLY the cohesion context. To keep the contrast carryover-ONLY (no backend-vs-rig confound), the
|
||||
rig generates BOTH a rig-A0 baseline (no context) and the Q3 arms; rig-A0 is validated to fall within the
|
||||
backend flat-A0 <-> flat-A0' noise floor.
|
||||
|
||||
Modes (§B3 / §D2 Q3a/Q3a'/Q3b):
|
||||
a0 : rig baseline, no cohesion context (bridge to backend flat-A0)
|
||||
q3a-src : + carryover = last K=3 SOURCE sentences of prev chunk (parallelism-safe)
|
||||
q3a-draft : + carryover = prev src tail ∥ prev chunk DRAFT tail (what the wave model gives)
|
||||
q3a-final : + carryover = prev src tail ∥ prev chunk FINAL tail (target-side, sequential pipeline)
|
||||
q3a-state : q3a-final + deterministic scene-state (active entities + their dst + gender from seed)
|
||||
q3b-look : + lookahead = first M=2 SOURCE sentences of NEXT chunk (+ seam-dedup guard in scoring)
|
||||
|
||||
Prompt assembly mirrors render.go: system(tpl.System + "\n\n" + FewShot) [CacheBoundary] -> system(injection)
|
||||
-> [system(cohesion context, read-only)] -> user(filled). Draft deepseek-v4-flash temp0.3 thinking-ON;
|
||||
editor glm-5 temp0.4 thinking-OFF. Per-call gate + usage/cost ledger (D30.10); resumable (skip existing).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import exp15_llm as L
|
||||
from mem_select import (build_bank, render_glossary_block, render_editor_constraint_block, union_sticky)
|
||||
from chunker import greedy_chunks, split_source_sentences
|
||||
|
||||
BACKEND = Path("/home/ubuntu/projects/textmachine/backend")
|
||||
S2 = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime_big_flat.txt") # 30-节 flat slice
|
||||
SEED = "/home/ubuntu/books/gu-zhenren/guzhenren-seed-v2.yaml"
|
||||
OUTDIR = Path("/home/ubuntu/books/gu-zhenren/exp15/arms")
|
||||
# Q3 baseline + carryover source = the REAL backend flat-A0-30 (not a rig-a0). Contrast = carryover-only.
|
||||
BASE_DB = "/home/ubuntu/books/gu-zhenren/exp15/anchors/flat_a0_30.db"
|
||||
BUDGET_TOK = 800
|
||||
K_CARRY = 3
|
||||
M_LOOK = 2
|
||||
|
||||
BRIEF = {"book_id": "exp15", "title": "蛊真人", "source_lang": "zh", "target_lang": "ru",
|
||||
"genre": "вебновелла", "audience": "взрослые читатели вебновелл", "venuti": "0.60",
|
||||
"honorifics": "keep", "transcription": "palladius", "footnotes": "minimal"}
|
||||
|
||||
CTX_HEADER = ("КОНТЕКСТ ДЛЯ СВЯЗНОСТИ (предыдущий/следующий фрагмент, ТОЛЬКО ДЛЯ ПОНИМАНИЯ переходов "
|
||||
"и разрешения местоимений/времён/рода — его НЕ переводить и НЕ включать в вывод):")
|
||||
STATE_HEADER = "АКТИВНЫЕ СУЩНОСТИ СЦЕНЫ (канонический перевод и род — для согласования местоимений/времён):"
|
||||
|
||||
|
||||
def parse_template(path):
|
||||
raw = Path(path).read_text(encoding="utf-8")
|
||||
parts = raw.split("---USER---", 1)
|
||||
head, user = parts[0], parts[1].strip()
|
||||
hp = head.split("---FEWSHOT---", 1)
|
||||
system = hp[0].strip()
|
||||
fewshot = hp[1].strip() if len(hp) == 2 else ""
|
||||
return {"system": system, "fewshot": fewshot, "user": user}
|
||||
|
||||
|
||||
def fill(tpl_text, **extra):
|
||||
vals = dict(BRIEF)
|
||||
vals.update(extra)
|
||||
out, rest = [], tpl_text
|
||||
while True:
|
||||
i = rest.find("{{")
|
||||
if i < 0:
|
||||
out.append(rest); break
|
||||
out.append(rest[:i]); rest = rest[i:]
|
||||
e = rest.find("}}")
|
||||
name = rest[2:e]
|
||||
out.append(vals[name]); rest = rest[e + 2:]
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def system_msg(tpl):
|
||||
return tpl["system"] + ("\n\n" + tpl["fewshot"] if tpl["fewshot"] else "")
|
||||
|
||||
|
||||
def seed_entities():
|
||||
import yaml
|
||||
d = yaml.safe_load(Path(SEED).read_text(encoding="utf-8"))
|
||||
return d["terms"]
|
||||
|
||||
|
||||
def scene_state_block(chunk_src, terms):
|
||||
"""Deterministic scene-state: seed entities present in the chunk, with dst + gender (§B3 state slot)."""
|
||||
lines = []
|
||||
seen = set()
|
||||
for t in terms:
|
||||
src = t.get("src"); dst = (t.get("dst") or "").strip()
|
||||
if not src or not dst:
|
||||
continue
|
||||
variants = [src] + [a.get("alias") for a in (t.get("aliases") or []) if a.get("alias")]
|
||||
if not any(v in chunk_src for v in variants):
|
||||
continue
|
||||
if t.get("type") not in ("name", "nickname"):
|
||||
continue
|
||||
g = t.get("gender") or ""
|
||||
gtag = {"male": "муж.", "female": "жен.", "hidden": "?", "": ""}.get(g, "")
|
||||
key = dst
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
lines.append(f"- {dst}" + (f" ({gtag} род)" if gtag else ""))
|
||||
return (STATE_HEADER + "\n" + "\n".join(lines)) if lines else ""
|
||||
|
||||
|
||||
def tail_sentences(text, k):
|
||||
s = split_source_sentences(text)
|
||||
return "".join(s[-k:]).strip() if s else ""
|
||||
|
||||
|
||||
def head_sentences(text, m):
|
||||
s = split_source_sentences(text)
|
||||
return "".join(s[:m]).strip() if s else ""
|
||||
|
||||
|
||||
def draft_context(mode, i, chunk_srcs, terms):
|
||||
"""SOURCE-side context for the DRAFT/translator (§B3-бис wave 1): src-carryover + scene-state + lookahead.
|
||||
The THREE q3a modes SHARE the src-tail into the draft; they differ only in the editor's target-side."""
|
||||
blocks = []
|
||||
if mode.startswith("q3a") and i > 0:
|
||||
blocks.append(CTX_HEADER + f"\n[предыдущий фрагмент, источник]:\n{tail_sentences(chunk_srcs[i-1], K_CARRY)}")
|
||||
if mode == "q3a-state":
|
||||
st = scene_state_block(chunk_srcs[i], terms)
|
||||
if st:
|
||||
blocks.append(st)
|
||||
if mode == "q3b-look" and i < len(chunk_srcs) - 1:
|
||||
blocks.append(CTX_HEADER + f"\n[следующий фрагмент, источник]:\n{head_sentences(chunk_srcs[i+1], M_LOOK)}")
|
||||
return "\n\n".join(blocks)
|
||||
|
||||
|
||||
def editor_context(mode, i, drafts, finals):
|
||||
"""TARGET-side context for the EDITOR (§B3-бис wave 2): prev neighbor's draft (wave) or final
|
||||
(sequential) tail. q3a-src has NONE (source-carryover baseline); q3b-look targets the draft only."""
|
||||
if i == 0:
|
||||
return ""
|
||||
if mode == "q3a-draft" and drafts and drafts.get(i - 1):
|
||||
return CTX_HEADER + f"\n[предыдущий фрагмент, черновой перевод соседа]:\n{tail_sentences(drafts[i-1], K_CARRY)}"
|
||||
if mode in ("q3a-final", "q3a-state") and finals and finals.get(i - 1):
|
||||
return CTX_HEADER + f"\n[предыдущий фрагмент, итоговый перевод соседа]:\n{tail_sentences(finals[i-1], K_CARRY)}"
|
||||
return ""
|
||||
|
||||
|
||||
def _reject(r):
|
||||
txt = (r.get("text") or "").strip()
|
||||
bad = bool(r.get("err")) or not txt or r.get("finish") not in ("stop", None)
|
||||
return txt, bad
|
||||
|
||||
|
||||
def run_q3a(sp, bank, terms, chunk_srcs, base_drafts, base_finals, caps):
|
||||
"""Q3a carrying block: ONE shared src-carryover DRAFT per chunk (§B3-бис: source-side -> wave-1 draft),
|
||||
then THREE editors differing ONLY in target-side neighbor context:
|
||||
q3a-src : no target context (source-carryover only)
|
||||
q3a-draft : flat-A0 prev-DRAFT tail (wave / draft-neighbor)
|
||||
q3a-final : flat-A0 prev-FINAL tail (sequential / final-neighbor)
|
||||
Sharing the draft removes the draft-noise confound between the 3 modes. Baseline = flat-A0-30 (backend).
|
||||
Resume-safe (skip non-empty existing files). Neighbors come from flat-A0-30 (parallel wave, no seq dep)."""
|
||||
tr = parse_template(BACKEND / "prompts/translator.md")
|
||||
ed = parse_template(BACKEND / "prompts/editor.md")
|
||||
tr_sys, ed_sys = fill(system_msg(tr)), fill(system_msg(ed))
|
||||
shared_dir = OUTDIR / "q3a-shared-draft"
|
||||
shared_dir.mkdir(parents=True, exist_ok=True)
|
||||
editor_modes = ["q3a-src", "q3a-draft", "q3a-final"]
|
||||
for m in editor_modes:
|
||||
(OUTDIR / m).mkdir(parents=True, exist_ok=True)
|
||||
active_ids = []
|
||||
limit = caps.get("max_chunks") or len(chunk_srcs)
|
||||
for i, src in enumerate(chunk_srcs):
|
||||
if i >= limit:
|
||||
break
|
||||
sticky = union_sticky(active_ids, len(active_ids))
|
||||
sel = bank.select(src, chapter=1, sticky_prev=sticky, budget_tokens=BUDGET_TOK)
|
||||
active_ids.append(sel.active_ids)
|
||||
gloss = render_glossary_block(sel.injected)
|
||||
edconstr = render_editor_constraint_block(sel.injected)
|
||||
dctx = draft_context("q3a-src", i, chunk_srcs, terms) # src-carryover (SHARED across the 3 modes)
|
||||
|
||||
# --- shared src-carryover draft (deepseek, temp 0.3), generated ONCE ---
|
||||
dpath = shared_dir / f"{i:02d}.draft.txt"
|
||||
if dpath.exists() and dpath.read_text(encoding="utf-8").strip():
|
||||
draft = dpath.read_text(encoding="utf-8")
|
||||
else:
|
||||
msgs = [{"role": "system", "content": tr_sys}]
|
||||
if gloss:
|
||||
msgs.append({"role": "system", "content": gloss})
|
||||
if dctx:
|
||||
msgs.append({"role": "system", "content": dctx})
|
||||
msgs.append({"role": "user", "content": fill(tr["user"], text=src)})
|
||||
ok, pred, why = sp.gate("deepseek-v4-flash", sum(len(m["content"]) for m in msgs) // 2, caps["draft_max"])
|
||||
if not ok:
|
||||
print(f" [gate BLOCK] shared draft ch{i}: {why}"); return
|
||||
r = L.call("deepseek-v4-flash", msgs, temp=0.3, max_tokens=caps["draft_max"])
|
||||
sp.record({"mode": "q3a-shared", "stage": "draft", "chunk": i, "usage": r.get("usage"), "cost": r.get("cost", 0.0), "err": r.get("err"), "finish": r.get("finish")})
|
||||
draft, bad = _reject(r)
|
||||
if bad:
|
||||
print(f" [BAD DRAFT] ch{i}: err={r.get('err')} finish={r.get('finish')}"); return
|
||||
dpath.write_text(draft, encoding="utf-8")
|
||||
|
||||
# --- three editors on the shared draft (glm-5, temp 0.4), differing only in target context ---
|
||||
for mode in editor_modes:
|
||||
fpath = OUTDIR / mode / f"{i:02d}.final.txt"
|
||||
if fpath.exists() and fpath.read_text(encoding="utf-8").strip():
|
||||
continue
|
||||
ectx = editor_context(mode, i, base_drafts, base_finals) # target-side (none/prev-draft/prev-final)
|
||||
emsgs = [{"role": "system", "content": ed_sys}]
|
||||
if edconstr:
|
||||
emsgs.append({"role": "system", "content": edconstr})
|
||||
if ectx:
|
||||
emsgs.append({"role": "system", "content": ectx})
|
||||
emsgs.append({"role": "user", "content": fill(ed["user"], text=src, draft=draft)})
|
||||
ok, pred, why = sp.gate("glm-5", sum(len(m["content"]) for m in emsgs) // 2, caps["edit_max"])
|
||||
if not ok:
|
||||
print(f" [gate BLOCK] {mode} ch{i} edit: {why}"); return
|
||||
r = L.call("glm-5", emsgs, temp=0.4, max_tokens=caps["edit_max"])
|
||||
sp.record({"mode": mode, "stage": "edit", "chunk": i, "usage": r.get("usage"), "cost": r.get("cost", 0.0), "err": r.get("err"), "finish": r.get("finish")})
|
||||
final, bad = _reject(r)
|
||||
if bad:
|
||||
print(f" [BAD FINAL] {mode} ch{i}: err={r.get('err')} finish={r.get('finish')}"); return
|
||||
fpath.write_text(final, encoding="utf-8")
|
||||
print(f" ch{i} {mode}: final {len(final)}c cost cum ${sp.total:.4f}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--modes", nargs="+", required=False,
|
||||
help="a0 q3a-src q3a-draft q3a-final q3a-state q3b-look")
|
||||
ap.add_argument("--per-call-cap", type=float, default=0.05)
|
||||
ap.add_argument("--hard-cap", type=float, default=6.0)
|
||||
ap.add_argument("--max-chunks", type=int, default=0, help="sanity-gate: only first N chunks (0=all)")
|
||||
a = ap.parse_args()
|
||||
bank = build_bank(SEED)
|
||||
terms = seed_entities()
|
||||
chunk_srcs = greedy_chunks(S2.read_text(encoding="utf-8"))
|
||||
|
||||
def load_base():
|
||||
"""Carryover source + Q3 baseline = REAL backend flat-A0-30 (draft + editor final per chunk)."""
|
||||
import sqlite3
|
||||
con = sqlite3.connect(BASE_DB)
|
||||
d, f = {}, {}
|
||||
for ck, txt in con.execute("SELECT chunk_idx, response_text FROM checkpoints WHERE stage='draft' ORDER BY chunk_idx, attempt"):
|
||||
d[ck] = txt
|
||||
for ck, txt in con.execute("SELECT chunk_idx, response_text FROM checkpoints WHERE stage='edit' ORDER BY chunk_idx, attempt"):
|
||||
f[ck] = txt
|
||||
con.close()
|
||||
return d, f
|
||||
|
||||
sp = L.Spender("/home/ubuntu/books/gu-zhenren/exp15/arm_costs.jsonl",
|
||||
per_call_cap=a.per_call_cap, hard_cap=a.hard_cap)
|
||||
caps = {"draft_max": 16000, "edit_max": 12000, "max_chunks": a.max_chunks}
|
||||
base_d, base_f = load_base()
|
||||
n_needed = len(chunk_srcs)
|
||||
if len(base_d) < n_needed or len(base_f) < n_needed:
|
||||
print(f" [ABORT] flat-A0-30 INCOMPLETE: draft {len(base_d)}/{n_needed}, final {len(base_f)}/{n_needed} "
|
||||
f"— the Q3 baseline + carryover source must be fully generated (regen must finish) first.")
|
||||
return
|
||||
lim = f", first {a.max_chunks} (sanity-gate)" if a.max_chunks else ""
|
||||
print(f"=== Q3a arms: shared src-carryover draft -> 3 editors (src/draft/final) over {len(chunk_srcs)} chunks{lim} ===")
|
||||
run_q3a(sp, bank, terms, chunk_srcs, base_d, base_f, caps)
|
||||
print(f"\narm spend this run: ${sp.total:.4f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
111
eval/exp15/assemble_traps.py
Normal file
111
eval/exp15/assemble_traps.py
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — assemble the FINAL 30-节 trap set: merge deterministic straddles (straddle_ledger.json:
|
||||
23 T-ent control + 7 T-ana) with kimi enrichment (ana_enrichment.json: T-ana/T-ell), dedup by
|
||||
(type, boundary), require each to STRADDLE a flat-A0-30 greedy cut (common denominator), and map each
|
||||
to the dependent chunk (i+1) so the judge can score each arm's rendering of that chunk.
|
||||
|
||||
Output final_traps.json: the trap set for Q1b (oracle vs flat) and Q3a (carryover) judging.
|
||||
Run: python assemble_traps.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from chunker import greedy_chunks
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
FLAT = SD / "S2prime_big_flat.txt"
|
||||
OUT = SD / "final_traps.json"
|
||||
|
||||
|
||||
def flat_boundaries(text):
|
||||
ch = greedy_chunks(text)
|
||||
offs, starts, pos = [], [0], 0
|
||||
for i, c in enumerate(ch):
|
||||
idx = text.find(c[:25], max(0, pos - 5))
|
||||
if idx < 0:
|
||||
idx = text.find(c[:25])
|
||||
if i > 0 and idx >= 0:
|
||||
offs.append(idx); starts.append(idx)
|
||||
pos = idx + len(c) if idx >= 0 else pos + len(c)
|
||||
return ch, offs, starts
|
||||
|
||||
|
||||
def chunk_of_offset(starts, off):
|
||||
c = 0
|
||||
for i, s in enumerate(starts):
|
||||
if off >= s:
|
||||
c = i
|
||||
return c
|
||||
|
||||
|
||||
def straddled_boundary(text, offs, a_span, d_span):
|
||||
ai = text.find(a_span[:25]); di = text.find(d_span[:25])
|
||||
if ai < 0 or di < 0:
|
||||
return None, None, None
|
||||
lo, hi = min(ai, di), max(ai + len(a_span), di + len(d_span))
|
||||
crossed = [k for k, cut in enumerate(offs) if lo < cut < hi]
|
||||
if not crossed:
|
||||
return None, ai, di
|
||||
return crossed[0], ai, di # boundary index (0-based cut = between chunk k and k+1)
|
||||
|
||||
|
||||
def main():
|
||||
text = FLAT.read_text(encoding="utf-8")
|
||||
ch, offs, starts = flat_boundaries(text)
|
||||
traps = []
|
||||
seen = set() # (type, boundary_cut)
|
||||
|
||||
def add(t, typ, control, a_span, d_span, phen, exp):
|
||||
bcut, ai, di = straddled_boundary(text, offs, a_span, d_span)
|
||||
if bcut is None:
|
||||
return "not_straddling"
|
||||
key = (typ, bcut)
|
||||
if key in seen:
|
||||
return "dup"
|
||||
seen.add(key)
|
||||
dep_chunk = chunk_of_offset(starts, di) # chunk containing the dependent (scored chunk)
|
||||
anc_chunk = chunk_of_offset(starts, ai)
|
||||
traps.append({"id": f"{typ}-{len(traps)+1}", "type": typ, "control": control,
|
||||
"boundary_cut": bcut, "anc_chunk": anc_chunk, "dep_chunk": dep_chunk,
|
||||
"anchor_span": a_span, "dependent_span": d_span,
|
||||
"phenomenon": phen, "expected_ru": exp})
|
||||
return "ok"
|
||||
|
||||
stats = {"ok": 0, "dup": 0, "not_straddling": 0}
|
||||
# deterministic
|
||||
det = json.load(open(SD / "straddle_ledger.json"))["traps"]
|
||||
for t in det:
|
||||
r = add(t, t["type"], t.get("control", False), t["anchor_span"], t["dependent_span"],
|
||||
t["phenomenon"], t["expected_ru"])
|
||||
stats[r] += 1
|
||||
# enrichment
|
||||
enr = json.load(open(SD / "ana_enrichment.json"))["straddles"]
|
||||
for t in enr:
|
||||
r = add(t, t.get("type", "T-ana"), False, t.get("antecedent_span", ""), t.get("dependent_span", ""),
|
||||
t.get("phenomenon", ""), t.get("expected_ru", ""))
|
||||
stats[r] += 1
|
||||
|
||||
from collections import Counter
|
||||
by = Counter(t["type"] for t in traps)
|
||||
prim = Counter(t["type"] for t in traps if not t["control"])
|
||||
OUT.write_text(json.dumps({
|
||||
"n": len(traps), "by_type": dict(by),
|
||||
"quota": {"T-ana": {"have": prim.get("T-ana", 0), "need": 20, "meets": prim.get("T-ana", 0) >= 20},
|
||||
"T-ent": {"have": by.get("T-ent", 0), "need": 20, "meets": by.get("T-ent", 0) >= 20, "role": "negative-control"},
|
||||
"T-ell": {"have": by.get("T-ell", 0), "need": 10, "meets": by.get("T-ell", 0) >= 10, "note": "secondary; short = honest-power"}},
|
||||
"undecidable_a_priori": ["T-tense", "T-cat", "T-gen", "T-reg"],
|
||||
"merge_stats": stats,
|
||||
"traps": traps,
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"final trap set: {dict(by)} (total {len(traps)}) merge {stats}")
|
||||
print(f" T-ana (primary): {prim.get('T-ana',0)} {'MEETS>=20' if prim.get('T-ana',0)>=20 else 'SHORT'}")
|
||||
print(f" T-ent (control): {by.get('T-ent',0)} {'MEETS>=20' if by.get('T-ent',0)>=20 else 'SHORT'}")
|
||||
print(f" T-ell (secondary): {by.get('T-ell',0)} (>=10? {by.get('T-ell',0)>=10})")
|
||||
print(f" undecidable a-priori: T-tense/T-cat/T-gen/T-reg -> {OUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
95
eval/exp15/blind_packets.py
Normal file
95
eval/exp15/blind_packets.py
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — BLIND reading packets for the owner (§D3 human endpoint: metrics NOMINATE, blind read RATIFIES,
|
||||
planка = <=2 claims/chapter). Presents the SAME opening source passage translated 3 ways — flat-A0-30
|
||||
(blind-greedy baseline), oracle-30 (seam-aligned), q3a-src (source-carryover arm) — under SHUFFLED neutral
|
||||
labels (V1/V2/V3), with a separate key. The owner reads without knowing which is which and counts claims.
|
||||
|
||||
Deterministic shuffle (fixed seed). Outputs to exp15/monitor/ (blind packet + key, key kept separate).
|
||||
Run: python blind_packets.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import random
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from chunker import greedy_chunks
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
AH = SD / "anchors"
|
||||
MON = SD / "monitor"
|
||||
N_CHUNKS = 6 # ~first 3-4 节 of the opening (readable passage)
|
||||
|
||||
|
||||
def db_finals(db):
|
||||
con = sqlite3.connect(db)
|
||||
out = {}
|
||||
for ck, txt in con.execute("SELECT chunk_idx, response_text FROM checkpoints WHERE stage='edit' ORDER BY chunk_idx, attempt"):
|
||||
out[ck] = txt
|
||||
con.close()
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
MON.mkdir(parents=True, exist_ok=True)
|
||||
flat_src = greedy_chunks((SD / "S2prime_big_flat.txt").read_text(encoding="utf-8"))
|
||||
oracle_src = greedy_chunks((SD / "S2prime_big_blanklines.txt").read_text(encoding="utf-8"))
|
||||
flat_fin = db_finals(AH / "flat_a0_30.db")
|
||||
oracle_fin = db_finals(AH / "oracle_30.db")
|
||||
|
||||
# source span covered by the first N flat chunks (used for oracle coverage matching AND now shown as
|
||||
# the fidelity reference — the reader compares each version against this original for omissions/errors)
|
||||
src_span = "".join(flat_src[:N_CHUNKS])
|
||||
src_display = "\n\n".join(flat_src[:N_CHUNKS]) # paragraph-joined for readability
|
||||
# flat + q3a-src share flat chunking; oracle differs -> take oracle chunks until it covers ~the same source len
|
||||
flat_text = "\n\n".join(flat_fin[i] for i in range(N_CHUNKS) if i in flat_fin)
|
||||
arm_dir = SD / "arms" / "q3a-src"
|
||||
src_arm = "\n\n".join((arm_dir / f"{i:02d}.final.txt").read_text(encoding="utf-8")
|
||||
for i in range(N_CHUNKS) if (arm_dir / f"{i:02d}.final.txt").exists())
|
||||
# oracle: chunks whose cumulative source length ~covers src_span
|
||||
cov, oi, otexts = 0, 0, []
|
||||
target = len(src_span)
|
||||
while oi < len(oracle_src) and cov < target:
|
||||
if oi in oracle_fin:
|
||||
otexts.append(oracle_fin[oi])
|
||||
cov += len(oracle_src[oi]); oi += 1
|
||||
oracle_text = "\n\n".join(otexts)
|
||||
|
||||
arms = {"flat-A0-30 (blind greedy baseline)": flat_text,
|
||||
"oracle-30 (seam-aligned)": oracle_text,
|
||||
"q3a-src (source-carryover)": src_arm}
|
||||
labels = ["V1", "V2", "V3"]
|
||||
keys = list(arms.keys())
|
||||
rnd = random.Random("exp15-blind-2026-07-18")
|
||||
rnd.shuffle(keys)
|
||||
mapping = dict(zip(labels, keys)) # V1->arm, ...
|
||||
|
||||
# blind packet (no arm names) — original source at top for FIDELITY comparison, then the 3 versions
|
||||
pkt = [f"# exp15 — СЛЕПОЙ ПАКЕТ для чтения (открывающий фрагмент 蛊真人, ~{N_CHUNKS} чанков)",
|
||||
"",
|
||||
"Три версии перевода ОДНОГО фрагмента под нейтральными метками (V1/V2/V3, перемешаны; ключ отдельно).",
|
||||
"Оцени: (а) художественность/связность и (б) ВЕРНОСТЬ — сверяя каждую версию с китайским ОРИГИНАЛОМ",
|
||||
"ниже (ничего не потеряно; смысл/род/референт/порядок событий переданы). Планка: ≤2 претензии на главу.",
|
||||
"Примечание: V1/V2/V3 переводились с разной нарезкой на чанки, поэтому у одной из версий хвост",
|
||||
"фрагмента может слегка отличаться по границе — это часть эксперимента, не ошибка перевода.",
|
||||
"", "---", "",
|
||||
"## ОРИГИНАЛ — китайский исходник (эталон для сверки верности)",
|
||||
"",
|
||||
src_display,
|
||||
"", "---", ""]
|
||||
for lab in labels:
|
||||
pkt += [f"## {lab}", "", arms[mapping[lab]], "", "---", ""]
|
||||
(MON / "blind_read_packet.md").write_text("\n".join(pkt), encoding="utf-8")
|
||||
(MON / "blind_read_KEY.json").write_text(json.dumps(
|
||||
{"_warning": "KEY — не показывать читающему до вынесения вердикта",
|
||||
"mapping_label_to_arm": mapping, "n_chunks": N_CHUNKS,
|
||||
"source_span_runes": len(src_span)}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"blind packet -> {MON}/blind_read_packet.md (key -> blind_read_KEY.json)")
|
||||
print(f" labels: {mapping}")
|
||||
print(f" lengths: flat {len(flat_text)}c, oracle {len(oracle_text)}c, q3a-src {len(src_arm)}c")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
139
eval/exp15/budget_calc.py
Normal file
139
eval/exp15/budget_calc.py
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — pre-reg budget recompute ($0), from live prices (freeze_facts.json).
|
||||
|
||||
Judge budget is recomputed FROM trap-N x votes x 2 orders x >=2 judges (prompt discipline: not the
|
||||
other way round). Hard cap $15 (owner). Per-call predicted-cost gate lives in the run scripts; this
|
||||
sizes the PACKAGE and picks a judge pair that fits. All assumptions are explicit + conservative so the
|
||||
freeze can pin them. Self-review: prints per-Q breakdown; asserts total logic.
|
||||
|
||||
Cost model (USD/1M tokens from freeze_facts.json):
|
||||
arm call = in_tok*price_in*(1-cache_frac) + in_tok*price_hit*cache_frac + out_tok*price_out
|
||||
judge call= in_tok*price_in + out_tok(+reasoning)*price_out (judges: no prefix cache assumed)
|
||||
Order of Qs (prompt): Q2b($0) -> Q1+Q3 (S2, carrying) -> Q2a/Q2c -> Q4 -> Q0. Budget likely covers
|
||||
the carrying block; later Qs run only if budget remains (pre-reg fixes priority, stop is legitimate).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
FACTS = json.load(open("/home/ubuntu/books/gu-zhenren/exp15/freeze_facts.json"))
|
||||
P = FACTS["prices"]
|
||||
|
||||
|
||||
def price(model, field, tier=0):
|
||||
v = P[model][field]
|
||||
return v[tier] if isinstance(v, list) else v
|
||||
|
||||
|
||||
def arm_call(model, in_tok, out_tok, cache_frac=0.0):
|
||||
pin = price(model, "in"); phit = P[model].get("cache_hit", pin); pout = price(model, "out")
|
||||
return (in_tok * pin * (1 - cache_frac) + in_tok * phit * cache_frac + out_tok * pout) / 1e6
|
||||
|
||||
|
||||
def judge_call(model, in_tok, out_tok):
|
||||
# reasoning/thinking billed as output for grok/gemini/gpt/kimi -> fold into out_tok by caller
|
||||
return (in_tok * price(model, "in") + out_tok * price(model, "out")) / 1e6
|
||||
|
||||
|
||||
# ---------------- ARM (generation) sizing ----------------
|
||||
# Conservative per-chunk token estimates (zh->ru; injection small; reflow-invariant).
|
||||
DRAFT_IN, DRAFT_OUT = 2600, 3200 # deepseek-v4-flash: src+small injection in; ru + thinking (billed out)
|
||||
EDIT_IN, EDIT_OUT = 5200, 2800 # glm-5: src+draft+injection in; ru edits out (thinking off for editor)
|
||||
CACHE_FRAC = 0.55 # static prefix (system+glossary) cached on sequential chunks (verified auto-cache)
|
||||
CHUNKS_PER_CH = 2.2 # ~2 greedy-1500 chunks per webnovel chapter (S2 est; S1 ~1/节)
|
||||
S2_CH = 10 # carrying-slice chapters (8-12)
|
||||
|
||||
def arm_cost_per_chapter(draft="deepseek-v4-flash", editor="glm-5"):
|
||||
per_chunk = arm_call(draft, DRAFT_IN, DRAFT_OUT, CACHE_FRAC) + arm_call(editor, EDIT_IN, EDIT_OUT, CACHE_FRAC)
|
||||
return per_chunk * CHUNKS_PER_CH
|
||||
|
||||
# Arms that GENERATE (Q0 do-nothing = $0 export; Q3 modes share one draft, differ in editor-context):
|
||||
ARMS_GEN = {
|
||||
"A0": 1, "A0'": 1, # double anchor
|
||||
"Q1-logical": 1,
|
||||
"Q2a-editGLava": 1, "Q2c-wideread": 1,
|
||||
"Q3a-src": 1, "Q3a-draft": 1, "Q3a-final": 1, "Q3a'-state": 1, "Q3b-lookahead": 1,
|
||||
"Q4a-strongdraft(v4pro)": 1, "Q4b-diff": 1, "Q4c-reflow": 1,
|
||||
}
|
||||
|
||||
# ---------------- JUDGE sizing (recompute from trap-N) ----------------
|
||||
# Trap set (common denominator, defined in A0). Minimums per prompt D1.
|
||||
TRAP_N = {"T-ana": 20, "T-ent": 20, "T-tense": 20, "T-cat": 10, "T-ell": 10,
|
||||
"T-gen": 10, "T-reg": 10, "T-inv": 10}
|
||||
# Which Qs judge which trap types, and #arm-contrasts each Q compares against A0 (pairwise A0-vs-arm):
|
||||
Q_JUDGE = {
|
||||
"Q1": (["T-tense", "T-ana"], 1), # A0 vs logical
|
||||
"Q3a": (["T-ana", "T-ell"], 3), # A0 vs {src,draft,final}
|
||||
"Q3a'": (["T-gen", "T-reg"], 1), # A0 vs +state
|
||||
"Q3b": (["T-cat"], 1), # A0 vs +lookahead
|
||||
# later Qs (run only if budget remains):
|
||||
"Q2c": (["T-ana", "T-tense"], 1),
|
||||
"Q4a": (["T-inv"], 1),
|
||||
"Q4b": (["T-inv"], 1),
|
||||
}
|
||||
CARRYING = {"Q1", "Q3a", "Q3a'", "Q3b"}
|
||||
VOTES_BASE = 6 # 3 per order x 2 orders
|
||||
ESCALATE_FRAC = 0.35 # fraction of cells that split -> +4 votes to 10
|
||||
JUDGES = ["grok-4.3", "mistral-large-latest"] # 2 cross-family; family-clean vs DeepSeek/GLM arms + kimi miner
|
||||
JUDGE_IN, JUDGE_OUT = {"grok-4.3": 1500, "mistral-large-latest": 1500}, {"grok-4.3": 900, "mistral-large-latest": 500}
|
||||
# grok reasoning ON (additive) -> larger out; mistral no reasoning surcharge.
|
||||
|
||||
def votes_expected():
|
||||
return VOTES_BASE + ESCALATE_FRAC * 4 # ~7.4
|
||||
|
||||
def judge_cost_for(qset, judges, jin, jout):
|
||||
total = 0.0; breakdown = {}
|
||||
for q, (types, contrasts) in Q_JUDGE.items():
|
||||
if q not in qset:
|
||||
continue
|
||||
traps = sum(TRAP_N[t] for t in types)
|
||||
cells = traps * contrasts # pairwise (A0-vs-arm) trap cells
|
||||
calls_per_judge = cells * votes_expected()
|
||||
c = 0.0
|
||||
for j in judges:
|
||||
c += calls_per_judge * judge_call(j, jin[j], jout[j])
|
||||
breakdown[q] = (traps, contrasts, cells, round(c, 2))
|
||||
total += c
|
||||
return total, breakdown
|
||||
|
||||
# ---------------- MINER (trap mining, paid) ----------------
|
||||
# kimi-k2.6 mines candidate cohesion phenomena over the S2 source; verbose reasoning billed as output.
|
||||
MINER = "kimi-k2.6"
|
||||
MINER_CALLS = S2_CH # ~1 mining pass per chapter
|
||||
MINER_IN, MINER_OUT = 3000, 6000 # chapter source in; candidate list + verbose reasoning out
|
||||
miner_cost = MINER_CALLS * judge_call(MINER, MINER_IN, MINER_OUT)
|
||||
|
||||
|
||||
def main():
|
||||
per_ch = arm_cost_per_chapter()
|
||||
arm_total = sum(ARMS_GEN.values()) * S2_CH * per_ch
|
||||
carrying_arm = 6 * S2_CH * per_ch # A0,A0',Q1,Q3a(counts as 1 draft+3 editor-ctx ~ approx 3),Q3a',Q3b ~6 arm-equivs
|
||||
j_all, jb = judge_cost_for(set(Q_JUDGE), JUDGES, JUDGE_IN, JUDGE_OUT)
|
||||
j_carry, jbc = judge_cost_for(CARRYING, JUDGES, JUDGE_IN, JUDGE_OUT)
|
||||
|
||||
print("=== exp15 budget recompute (live prices, 2026-07-17) ===")
|
||||
print(f"arm cost / chapter (draft deepseek-v4-flash + editor glm-5, cache_frac={CACHE_FRAC}): ${per_ch:.4f}")
|
||||
print(f" draft/chunk ${arm_call('deepseek-v4-flash',DRAFT_IN,DRAFT_OUT,CACHE_FRAC):.4f} editor/chunk ${arm_call('glm-5',EDIT_IN,EDIT_OUT,CACHE_FRAC):.4f} chunks/ch {CHUNKS_PER_CH}")
|
||||
print(f"trap miner ({MINER}, {MINER_CALLS} calls): ${miner_cost:.2f}")
|
||||
print(f"expected votes/cell: {votes_expected():.1f} judges: {JUDGES}")
|
||||
print("\n-- judge cost per Q (traps, contrasts, cells, $) --")
|
||||
for q, v in jb.items():
|
||||
tag = "CARRY" if q in CARRYING else "later"
|
||||
print(f" [{tag}] {q}: traps={v[0]} contrasts={v[1]} cells={v[2]} -> ${v[3]}")
|
||||
print(f"\nCARRYING block: arms ${carrying_arm:.2f} + judges ${j_carry:.2f} + miner ${miner_cost:.2f} = ${carrying_arm+j_carry+miner_cost:.2f}")
|
||||
print(f"FULL package (all listed Qs): arms ${arm_total:.2f} + judges ${j_all:.2f} + miner ${miner_cost:.2f} = ${arm_total+j_all+miner_cost:.2f}")
|
||||
print(f"HARD CAP: $15.00")
|
||||
carry_total = carrying_arm + j_carry + miner_cost
|
||||
print(f"\nVERDICT: carrying block ${carry_total:.2f} {'FITS' if carry_total<=15 else 'EXCEEDS'} $15; "
|
||||
f"later Qs (Q2c/Q4) run only if budget remains after carrying block + per-call gate.")
|
||||
# sensitivity: apex secondary judge (gemini mandatory thinking billed as output)
|
||||
apex_judges = ["grok-4.3", "gemini-3.1-pro-preview"]
|
||||
apex_in = {**JUDGE_IN, "gemini-3.1-pro-preview": 1500}
|
||||
apex_out = {**JUDGE_OUT, "gemini-3.1-pro-preview": 2500}
|
||||
jg, _ = judge_cost_for(CARRYING, apex_judges, apex_in, apex_out)
|
||||
print(f"\n[sensitivity] if secondary judge = gemini-3.1-pro (apex, thinking billed out): carrying judges ${jg:.2f} "
|
||||
f"-> block ${carrying_arm+jg+miner_cost:.2f} ({'FITS' if carrying_arm+jg+miner_cost<=15 else 'EXCEEDS'} $15)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
113
eval/exp15/build_s2prime.py
Normal file
113
eval/exp15/build_s2prime.py
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — S2' synthetic scene-marked slice via guzhenren chapter concatenation (owner decision 2026-07-17).
|
||||
|
||||
S2' = concatenate the FIRST N consecutive 节 of 蛊真人 (opening arc), STRIP the 第X节 headers at the seams,
|
||||
so each seam is an IMPLICIT scene boundary (no explicit marker). Every seam is recorded as a ground-truth
|
||||
boundary (rune-offset in the concatenated text + the two adjacent 节 ids). The seam previews (tail of 节_i +
|
||||
head of 节_{i+1}) are emitted so each seam can be MANUALLY classified scene-shift vs continuation-cliffhanger
|
||||
(§1.14b): continuation seams are negative-boundaries ("must NOT cut here"), separate strata for Q1.
|
||||
|
||||
Deterministic, $0, pure function of the book bytes. Outputs OUTSIDE git.
|
||||
Run: eval/.venv/bin/python eval/exp15/build_s2prime.py [N] # default N=12 节
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
BOOK = Path("/home/ubuntu/books/gu-zhenren/guzhenren-utf8.txt")
|
||||
OUT_DIR = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
OUT_TXT = OUT_DIR / "S2prime.txt"
|
||||
OUT_SEAMS = OUT_DIR / "S2prime_seams.json"
|
||||
OUT_PREVIEW = OUT_DIR / "S2prime_seam_previews.md"
|
||||
|
||||
SEC_RE = re.compile(r"^第[0-9一二三四五六七八九十百千零两]{1,10}[节節][：: ]?.*$", re.M)
|
||||
PREVIEW_TAIL, PREVIEW_HEAD = 220, 220
|
||||
|
||||
|
||||
def load_text() -> str:
|
||||
raw = BOOK.read_bytes()
|
||||
try:
|
||||
return raw.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return raw.decode("gb18030", errors="replace")
|
||||
|
||||
|
||||
def sections(text: str):
|
||||
"""Return list of (header_line, body_text) for each 节 in file order (body = up to next 节)."""
|
||||
marks = list(SEC_RE.finditer(text))
|
||||
out = []
|
||||
for k, m in enumerate(marks):
|
||||
header = m.group(0).strip()
|
||||
body_start = m.end()
|
||||
body_end = marks[k + 1].start() if k + 1 < len(marks) else len(text)
|
||||
body = text[body_start:body_end].strip("\n")
|
||||
out.append((k + 1, header, body)) # sec index n = k+1 (running, per gu_corpus_build convention)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
n = int(sys.argv[1]) if len(sys.argv) > 1 else 12
|
||||
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
text = load_text()
|
||||
secs = sections(text)
|
||||
if len(secs) < n:
|
||||
print(f"only {len(secs)} 节 found; using all", file=sys.stderr)
|
||||
n = len(secs)
|
||||
chosen = secs[:n]
|
||||
|
||||
# concatenate bodies (headers STRIPPED at seams); single "\n" join = the implicit seam
|
||||
parts, seams, cursor = [], [], 0
|
||||
JOIN = "\n"
|
||||
for i, (idx, header, body) in enumerate(chosen):
|
||||
if i > 0:
|
||||
cursor += len(JOIN) # account the join separator (rune count)
|
||||
seam_offset = cursor # rune offset where this section's body begins
|
||||
if i > 0:
|
||||
seams.append({
|
||||
"seam_id": i,
|
||||
"rune_offset": seam_offset, # boundary between prev body and this body
|
||||
"prev_sec": chosen[i - 1][0],
|
||||
"next_sec": idx,
|
||||
"prev_header": chosen[i - 1][1],
|
||||
"next_header": header,
|
||||
"class": "UNCLASSIFIED", # -> scene-shift | continuation (manual, §1.14b)
|
||||
"rationale": "",
|
||||
})
|
||||
parts.append(body)
|
||||
cursor += len(body)
|
||||
joined = JOIN.join(parts)
|
||||
|
||||
OUT_TXT.write_text(joined, encoding="utf-8")
|
||||
OUT_SEAMS.write_text(json.dumps({
|
||||
"book": "guzhenren", "arc": "opening", "n_sections": n,
|
||||
"section_ids": [c[0] for c in chosen],
|
||||
"total_runes": len(joined),
|
||||
"headers_stripped_at_seams": True,
|
||||
"seam_marker": "implicit (single \\n; no 第X节 header)",
|
||||
"seams": seams,
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
# seam-context previews for MANUAL classification (read tail_i + head_{i+1})
|
||||
pv = [f"# S2' seam previews — classify each: scene-shift | continuation-cliffhanger (§1.14b)",
|
||||
f"# {n} 节 concatenated, headers stripped; {len(seams)} seams; total {len(joined):,} runes", ""]
|
||||
for s in seams:
|
||||
off = s["rune_offset"]
|
||||
tail = joined[max(0, off - len(JOIN) - PREVIEW_TAIL):off - len(JOIN)]
|
||||
head = joined[off:off + PREVIEW_HEAD]
|
||||
pv += [f"## seam {s['seam_id']} (节{s['prev_sec']} → 节{s['next_sec']}, offset {off})",
|
||||
f"**{s['prev_header']}** → **{s['next_header']}**",
|
||||
f"…tail of 节{s['prev_sec']}: …{tail}",
|
||||
f"head of 节{s['next_sec']}: {head}…", ""]
|
||||
OUT_PREVIEW.write_text("\n".join(pv), encoding="utf-8")
|
||||
|
||||
print(f"S2' built: {n} 节, {len(seams)} seams, {len(joined):,} runes -> {OUT_TXT}")
|
||||
print(f"section ids: {[c[0] for c in chosen]}")
|
||||
print(f"seams (offset, 节i->节j): " + ", ".join(f"{s['rune_offset']}({s['prev_sec']}->{s['next_sec']})" for s in seams))
|
||||
print(f"previews for manual classification -> {OUT_PREVIEW}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
65
eval/exp15/build_s2prime_big.py
Normal file
65
eval/exp15/build_s2prime_big.py
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — build the EXPANDED S2' (full-power, owner decision 2026-07-18) from the first N 节 of 蛊真人.
|
||||
|
||||
Produces TWO aligned versions of the same content (only the seam separator differs), + seam ledger:
|
||||
S2prime_big_flat.txt seams = single '\\n' -> greedy A0 is BLIND (blind-budget cuts)
|
||||
S2prime_big_blanklines.txt seams = '\\n\\n' -> greedy ORACLE cuts AT every seam (paragraph flush)
|
||||
Within a 节 the body is normalized to single '\\n' (collapse internal blank lines) so the ONLY '\\n\\n'
|
||||
are at seams — that is what makes the oracle cut exactly at scene boundaries.
|
||||
|
||||
Labels for seams are NOT hand-classified here (too many); Q1b uses the oracle's cut-at-all-seams behavior
|
||||
and the covariate is per-seam. Deterministic straddle-trap finding happens in straddle_traps.py.
|
||||
Run: python build_s2prime_big.py [N] # default N=45
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import build_s2prime as B
|
||||
|
||||
OUT = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
|
||||
|
||||
def main():
|
||||
n = int(sys.argv[1]) if len(sys.argv) > 1 else 45
|
||||
secs = B.sections(B.load_text())[:n]
|
||||
bodies = []
|
||||
for idx, header, body in secs:
|
||||
b = body.replace("\r\n", "\n").replace("\r", "\n")
|
||||
b = re.sub(r"\n{2,}", "\n", b).strip("\n") # single-\n within 节
|
||||
bodies.append((idx, b))
|
||||
flat = "\n".join(b for _, b in bodies)
|
||||
blank = "\n\n".join(b for _, b in bodies)
|
||||
|
||||
# seam offsets in each version (start of body i, i>0)
|
||||
def seam_offsets(sep):
|
||||
offs, cur = [], 0
|
||||
for i, (idx, b) in enumerate(bodies):
|
||||
if i > 0:
|
||||
cur += len(sep)
|
||||
if i > 0:
|
||||
offs.append({"seam_id": i, "prev_sec": bodies[i - 1][0], "next_sec": idx, "rune_offset": cur})
|
||||
cur += len(b)
|
||||
return offs
|
||||
|
||||
(OUT / "S2prime_big_flat.txt").write_text(flat, encoding="utf-8")
|
||||
(OUT / "S2prime_big_blanklines.txt").write_text(blank, encoding="utf-8")
|
||||
(OUT / "S2prime_big_seams_flat.json").write_text(json.dumps(
|
||||
{"version": "flat", "n_sections": n, "section_ids": [b[0] for b in bodies],
|
||||
"total_runes": len(flat), "seams": seam_offsets("\n")}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
(OUT / "S2prime_big_seams_blank.json").write_text(json.dumps(
|
||||
{"version": "blanklines", "n_sections": n, "section_ids": [b[0] for b in bodies],
|
||||
"total_runes": len(blank), "seams": seam_offsets("\n\n")}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"S2'-big: {n}节 flat={len(flat)} runes blank={len(blank)} runes seams={n-1}")
|
||||
# verify seam alignment
|
||||
fs = seam_offsets("\n"); bs = seam_offsets("\n\n")
|
||||
ok = all(flat[s["rune_offset"]:s["rune_offset"]+3] == bodies[i+1][1][:3] for i, s in enumerate(fs))
|
||||
okb = all(blank[s["rune_offset"]:s["rune_offset"]+3] == bodies[i+1][1][:3] for i, s in enumerate(bs))
|
||||
print(f"seam alignment: flat={ok} blank={okb}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
441
eval/exp15/chunker.py
Normal file
441
eval/exp15/chunker.py
Normal file
|
|
@ -0,0 +1,441 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — chunkers for Q1: a faithful zh source-sentence splitter + the §B1.5 LOGICAL DP chunker.
|
||||
|
||||
Q1 contrast (design §D2): A0 (greedy-1500, from the real backend) vs a LOGICAL-boundary arm sized to
|
||||
A0's ACTUAL est_out distribution (±10% mean). The ONLY variable is cut ALIGNMENT (greedy is blind to
|
||||
scene structure; logical aligns cuts to scene-shift boundaries and avoids mid-scene/dialogue cuts).
|
||||
Size is held equal so Q1 does not confound boundary class with chunk size (that is Q2's domain).
|
||||
|
||||
Ports (backend/internal/pipeline/chunker.go):
|
||||
splitSourceSentences / isOpenQuote / isCloseQuote / isSourceTerminator / isCJKTerminator /
|
||||
isClosingBracket / isASCIISpace (chunker.go:196-357) — quote-depth-aware, lossless tiling.
|
||||
Token math:
|
||||
est_tokens = cjk + other//3 (EstimateTokens units — the greedy 1500 budget space)
|
||||
est_out = 1.1978*cjk + 0.3852*other (fertility_calib.json — glm-4.6 ru output tokens; §B1.2)
|
||||
|
||||
The logical chunker implements §B1.4 boundary classes (B0 chapter/separator, B1 scene-shift lexicon,
|
||||
B2 TextTiling cohesion valley, B3 paragraph, B4 sentence) and §B1.5 DP packing (Knuth-Plass shortest
|
||||
path over candidate cuts, INTEGER-scaled costs, asymmetric size penalty). It does NOT peek at the
|
||||
ground-truth seam offsets — scene-shifts are DISCOVERED (that is the Q1 test).
|
||||
|
||||
Self-review by execution: `python chunker.py --selftest` checks tiling losslessness + a size sweep.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import sys
|
||||
import unicodedata
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import regex
|
||||
|
||||
_RE_CJK = regex.compile(r"[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]")
|
||||
TARGET_CHUNK_TOKENS = 1500 # chunker.go:47 const
|
||||
|
||||
F_CJK, F_OTHER = 1.1978, 0.3852 # fertility_calib.json (glm-4.6 ru out tokens from src char classes)
|
||||
|
||||
# --- char-class token estimators -------------------------------------------------------------------
|
||||
def _classes(text: str):
|
||||
cjk = other = 0
|
||||
for ch in text:
|
||||
if _RE_CJK.match(ch):
|
||||
cjk += 1
|
||||
elif ch.isspace():
|
||||
pass
|
||||
else:
|
||||
other += 1
|
||||
return cjk, other
|
||||
|
||||
|
||||
def est_tokens(text: str) -> int:
|
||||
"""EstimateTokens units (source side): cjk + other//3, floored at 16 (chunker.go estTokensFrom)."""
|
||||
cjk, other = _classes(text)
|
||||
v = cjk + other // 3
|
||||
return max(v, 16)
|
||||
|
||||
|
||||
def est_out(text: str) -> float:
|
||||
"""Predicted glm-4.6 ru OUTPUT tokens for a source slice (fertility model; §B1.2)."""
|
||||
cjk, other = _classes(text)
|
||||
return F_CJK * cjk + F_OTHER * other
|
||||
|
||||
|
||||
# --- GREEDY chunker port (chunker.go:82-194 + render.go:285 NormalizeSource) -----------------------
|
||||
def normalize_source(s: str) -> str:
|
||||
"""render.go:285 — BOM strip, CRLF/CR->LF, valid-UTF8 (Python strs already are), NFC, TrimSpace."""
|
||||
s = s.lstrip("")
|
||||
s = s.replace("\r\n", "\n").replace("\r", "\n")
|
||||
s = unicodedata.normalize("NFC", s)
|
||||
return s.strip() # Go strings.TrimSpace strips Unicode whitespace incl. U+3000 (== Python str.strip())
|
||||
|
||||
|
||||
def _class_counts(s: str):
|
||||
cjk = other = 0
|
||||
for ch in s:
|
||||
if _RE_CJK.match(ch):
|
||||
cjk += 1
|
||||
elif ch.isspace():
|
||||
pass
|
||||
else:
|
||||
other += 1
|
||||
return cjk, other
|
||||
|
||||
|
||||
def _est_tokens_from(cjk, other):
|
||||
return max(cjk + other // 3, 16) # chunker.go estTokensFrom (integer // 3, floor 16)
|
||||
|
||||
|
||||
def split_paragraphs(chapter: str):
|
||||
return [p.strip() for p in chapter.split("\n\n") if p.strip()]
|
||||
|
||||
|
||||
def _pack_sentences(paragraph: str):
|
||||
segs = split_source_sentences(paragraph)
|
||||
chunks, buf, bc, bo = [], [], 0, 0
|
||||
def flush():
|
||||
nonlocal buf, bc, bo
|
||||
if buf:
|
||||
chunks.append("".join(buf)); buf, bc, bo = [], 0, 0
|
||||
for s in segs:
|
||||
sc, so = _class_counts(s)
|
||||
if buf and _est_tokens_from(bc + sc, bo + so) > TARGET_CHUNK_TOKENS:
|
||||
flush()
|
||||
buf.append(s); bc, bo = bc + sc, bo + so
|
||||
flush()
|
||||
return chunks
|
||||
|
||||
|
||||
def greedy_chunks(raw_text: str):
|
||||
"""Port of chunker.go SplitChunks for ONE chapter (S2' has no chapter markers). Returns chunk texts."""
|
||||
chapter = normalize_source(raw_text)
|
||||
paras = split_paragraphs(chapter)
|
||||
chunks, buf, bc, bo = [], [], 0, 0
|
||||
def flush():
|
||||
nonlocal buf, bc, bo
|
||||
if buf:
|
||||
text = "\n\n".join(buf).strip()
|
||||
if text:
|
||||
chunks.append(text)
|
||||
buf, bc, bo = [], 0, 0
|
||||
for p in paras:
|
||||
pc, po = _class_counts(p)
|
||||
if _est_tokens_from(pc, po) > TARGET_CHUNK_TOKENS:
|
||||
flush()
|
||||
for sub in _pack_sentences(p):
|
||||
t = sub.strip()
|
||||
if t:
|
||||
chunks.append(t)
|
||||
continue
|
||||
if buf and _est_tokens_from(bc + pc, bo + po) > TARGET_CHUNK_TOKENS:
|
||||
flush()
|
||||
buf.append(p); bc, bo = bc + pc, bo + po
|
||||
flush()
|
||||
return chunks
|
||||
|
||||
|
||||
# --- source sentence splitter (port of chunker.go:196-357) -----------------------------------------
|
||||
_OPEN_Q = set("「『（【《〈［｛“‘")
|
||||
_CLOSE_Q = set("」』）】》〉］｝”’")
|
||||
_TERM = set("。！？.!?…")
|
||||
_CJK_TERM = set("。！？")
|
||||
_CLOSE_BRK = set("」』】）)》〉］]｝}\"'”’")
|
||||
_ASCII_SPACE = set(" \t\n\r\f\v ")
|
||||
|
||||
|
||||
def split_source_sentences(paragraph: str):
|
||||
runes = list(paragraph)
|
||||
segs = []
|
||||
start = i = depth = 0
|
||||
n = len(runes)
|
||||
while i < n:
|
||||
r = runes[i]
|
||||
if r in _OPEN_Q:
|
||||
depth += 1
|
||||
i += 1
|
||||
continue
|
||||
if r in _CLOSE_Q:
|
||||
if depth > 0:
|
||||
depth -= 1
|
||||
i += 1
|
||||
continue
|
||||
if r not in _TERM:
|
||||
i += 1
|
||||
continue
|
||||
j = i
|
||||
cjk = False
|
||||
while j < n and runes[j] in _TERM:
|
||||
if runes[j] in _CJK_TERM:
|
||||
cjk = True
|
||||
j += 1
|
||||
depth_here = depth
|
||||
e = j
|
||||
while e < n and runes[e] in _CLOSE_BRK:
|
||||
if runes[e] in _CLOSE_Q and depth > 0:
|
||||
depth -= 1
|
||||
e += 1
|
||||
if cjk:
|
||||
boundary = depth_here == 0
|
||||
else:
|
||||
boundary = depth_here == 0 and (e >= n or runes[e] in _ASCII_SPACE)
|
||||
# abbrev guard omitted for zh source (CJK terminators dominate; en not in S2')
|
||||
if not boundary:
|
||||
i = e
|
||||
continue
|
||||
w = e
|
||||
while w < n and runes[w] in _ASCII_SPACE:
|
||||
w += 1
|
||||
segs.append("".join(runes[start:w]))
|
||||
start = i = w
|
||||
if start < n:
|
||||
segs.append("".join(runes[start:]))
|
||||
return segs
|
||||
|
||||
|
||||
# --- units: split the text into atomic sentence-ish spans that TILE it -----------------------------
|
||||
def units_from_text(text: str):
|
||||
"""Atomic units for the DP = source sentences. Preserves whether each unit ENDS a paragraph
|
||||
(\\n\\n) or a line (\\n) for boundary classification. Tiling: ''.join(u.text) == text."""
|
||||
out = []
|
||||
# split into paragraph blocks on blank lines, remember the separator that followed each block
|
||||
# (so paragraph-end vs line-end vs sentence-end is recoverable). We tile exactly by re-attaching
|
||||
# the separators to the preceding unit.
|
||||
for seg, sep in _iter_blocks(text):
|
||||
sents = split_source_sentences(seg)
|
||||
for k, s in enumerate(sents):
|
||||
ends_para = (k == len(sents) - 1) and sep == "\n\n"
|
||||
ends_line = (k == len(sents) - 1) and sep == "\n"
|
||||
out.append((s, ends_para, ends_line))
|
||||
# re-attach the block separator to the last unit so tiling is exact
|
||||
if out and sep:
|
||||
s, ep, el = out[-1]
|
||||
out[-1] = (s + sep, ep, el)
|
||||
return out
|
||||
|
||||
|
||||
def _iter_blocks(text: str):
|
||||
"""Yield (block_text, separator) where separator is '\\n\\n' | '\\n' | ''. Tiles the input."""
|
||||
i = 0
|
||||
n = len(text)
|
||||
buf_start = 0
|
||||
while i < n:
|
||||
if text[i] == "\n":
|
||||
if i + 1 < n and text[i + 1] == "\n":
|
||||
yield text[buf_start:i], "\n\n"
|
||||
i += 2
|
||||
buf_start = i
|
||||
continue
|
||||
else:
|
||||
yield text[buf_start:i], "\n"
|
||||
i += 1
|
||||
buf_start = i
|
||||
continue
|
||||
i += 1
|
||||
if buf_start < n:
|
||||
yield text[buf_start:], ""
|
||||
|
||||
|
||||
# --- boundary classification (§B1.4) ---------------------------------------------------------------
|
||||
# B1 scene-shift lexicon: STRONG time/place/POV shift markers in the first chars of the NEXT unit.
|
||||
# Curated generic zh markers (NOT the S2' seam text — scene-shifts must be DISCOVERED). Kept to strong
|
||||
# temporal/spatial jumps + narrative shift particles; weak mid-scene connectors (顿时/忽然) excluded.
|
||||
_B1_MARKERS = [
|
||||
# temporal jumps
|
||||
"次日", "翌日", "第二天", "第三天", "隔日", "数日后", "数天后", "几天后", "三天后", "三天三夜",
|
||||
"一个星期", "一周", "几个月", "半年", "一年后", "三年后", "五年后", "多年后",
|
||||
"当天", "这一天", "那一天", "这一日", "翌朝", "清晨", "早晨", "拂晓", "五更", "天亮", "天明",
|
||||
"傍晚", "黄昏", "夜里", "半夜", "深夜", "入夜", "朝阳", "红日", "夕阳", "日落", "日出", "正午",
|
||||
"转眼", "转眼间", "很快", "不久", "不多时", "没过多久", "光阴荏苒", "时光飞逝",
|
||||
# narrative-shift particles / POV or place switch
|
||||
"却说", "再说", "话说", "单说", "且说", "与此同时", "另一边", "另一方", "一方", "此时此刻",
|
||||
"回到", "镜头", "视线",
|
||||
]
|
||||
|
||||
|
||||
def _b1_scene_shift(next_unit_text: str) -> bool:
|
||||
head = next_unit_text.lstrip(" \n\t")[:12]
|
||||
return any(m in head for m in _B1_MARKERS)
|
||||
|
||||
|
||||
def _dialogue_boundary_penalty(prev_unit: str, next_unit: str) -> int:
|
||||
"""Penalize cutting between a reply and its attribution or inside a dialogue exchange (§A.6):
|
||||
if prev ends inside/with a quote and next continues speech attribution, cutting is bad."""
|
||||
p = prev_unit.rstrip(" \n\t")
|
||||
nx = next_unit.lstrip(" \n\t")
|
||||
pen = 0
|
||||
# prev ends with a closing quote AND next starts with attribution-ish (说道/道/问/答 ...) -> keep together
|
||||
if p and p[-1] in _CLOSE_Q and nx[:2] in ("说道", "说", "道", "问道", "答道", "喝道", "笑道"):
|
||||
pen += 1
|
||||
# next starts a quote while prev did not close one cleanly (mid-exchange) — mild
|
||||
if nx[:1] in _OPEN_Q and p and p[-1] in _OPEN_Q:
|
||||
pen += 1
|
||||
return pen
|
||||
|
||||
|
||||
# --- TextTiling cohesion valley (B2) ---------------------------------------------------------------
|
||||
def _char_bigrams(text: str):
|
||||
t = [c for c in text if not c.isspace()]
|
||||
return [t[k] + t[k + 1] for k in range(len(t) - 1)]
|
||||
|
||||
|
||||
def _cosine(a: dict, b: dict) -> float:
|
||||
if not a or not b:
|
||||
return 0.0
|
||||
dot = sum(v * b.get(k, 0) for k, v in a.items())
|
||||
na = math.sqrt(sum(v * v for v in a.values()))
|
||||
nb = math.sqrt(sum(v * v for v in b.values()))
|
||||
return dot / (na * nb) if na and nb else 0.0
|
||||
|
||||
|
||||
def _bag(units, lo, hi):
|
||||
d = {}
|
||||
for u in units[lo:hi]:
|
||||
for bg in _char_bigrams(u[0]):
|
||||
d[bg] = d.get(bg, 0) + 1
|
||||
return d
|
||||
|
||||
|
||||
def texttiling_depths(units, w=4):
|
||||
"""Depth score of the lexical-cohesion valley at each inter-unit gap (higher = stronger boundary).
|
||||
Sliding window of w units either side; cosine of char-bigram bags; depth = local valley prominence."""
|
||||
n = len(units)
|
||||
gaps = n - 1
|
||||
sims = [0.0] * gaps
|
||||
for g in range(gaps):
|
||||
left = _bag(units, max(0, g + 1 - w), g + 1)
|
||||
right = _bag(units, g + 1, min(n, g + 1 + w))
|
||||
sims[g] = _cosine(left, right)
|
||||
# depth score: for each gap, (peak_left - sim) + (peak_right - sim)
|
||||
depths = [0.0] * gaps
|
||||
for g in range(gaps):
|
||||
lpk = sims[g]
|
||||
k = g
|
||||
while k > 0 and sims[k - 1] >= sims[k]:
|
||||
k -= 1
|
||||
lpk = max(lpk, sims[k])
|
||||
rpk = sims[g]
|
||||
k = g
|
||||
while k < gaps - 1 and sims[k + 1] >= sims[k]:
|
||||
k += 1
|
||||
rpk = max(rpk, sims[k])
|
||||
depths[g] = (lpk - sims[g]) + (rpk - sims[g])
|
||||
return depths
|
||||
|
||||
|
||||
# --- §B1.5 DP logical chunker ----------------------------------------------------------------------
|
||||
@dataclass
|
||||
class ChunkPlan:
|
||||
chunks: list # list of (text, est_out, right_boundary_class)
|
||||
cut_offsets: list # rune offsets in the ORIGINAL text where a cut falls (chunk starts, excl 0)
|
||||
|
||||
|
||||
# boundary-class penalty (§B1.5): B0=0 < B1 < B2 < B3 < B4 (prefer cutting at strong boundaries)
|
||||
_CLASS_PEN = {"B0": 0, "B1": 1, "B2": 3, "B3": 6, "B4": 10}
|
||||
|
||||
|
||||
def logical_chunks(text: str, target_out: float, hard_max_out: float, min_out: float = None,
|
||||
w_size=3, w_bound=60, w_dialog=200, b2_z=0.8,
|
||||
corridor_lo=0.62, corridor_hi=1.12) -> ChunkPlan:
|
||||
"""§B1.5 DP: minimize Σ [w_size·size_pen + w_bound·class_pen(right boundary) + w_dialog·dialogue].
|
||||
Cuts only at unit (sentence) boundaries — never mid-sentence (B5 forbidden). Costs INTEGER-scaled.
|
||||
target_out/hard_max_out = A0's actual est_out mean/max (size-match)."""
|
||||
units = units_from_text(text)
|
||||
n = len(units)
|
||||
if min_out is None:
|
||||
min_out = 0.45 * target_out
|
||||
# per-unit est_out and cumulative rune offsets
|
||||
out_of = [est_out(u[0]) for u in units]
|
||||
cum_out = [0.0] * (n + 1)
|
||||
cum_rune = [0] * (n + 1)
|
||||
for k in range(n):
|
||||
cum_out[k + 1] = cum_out[k] + out_of[k]
|
||||
cum_rune[k + 1] = cum_rune[k] + len(units[k][0])
|
||||
|
||||
# boundary class at each inter-unit gap g (cut AFTER unit g, i.e. between unit g and g+1)
|
||||
depths = texttiling_depths(units)
|
||||
if depths:
|
||||
mu = sum(depths) / len(depths)
|
||||
sd = math.sqrt(sum((d - mu) ** 2 for d in depths) / len(depths)) or 1e-9
|
||||
gap_class = []
|
||||
gap_dialog = []
|
||||
for g in range(n - 1):
|
||||
cls = "B4" # default: sentence boundary
|
||||
if units[g][2] or units[g][1]:
|
||||
cls = "B3" # ends a line/paragraph
|
||||
if depths and (depths[g] - mu) / sd >= b2_z:
|
||||
cls = "B2" # TextTiling cohesion valley
|
||||
if _b1_scene_shift(units[g + 1][0]):
|
||||
cls = "B1" # scene-shift lexicon in next unit
|
||||
gap_class.append(cls)
|
||||
gap_dialog.append(_dialogue_boundary_penalty(units[g][0], units[g + 1][0]))
|
||||
|
||||
lo_band, hi_band = corridor_lo * target_out, corridor_hi * target_out
|
||||
def size_pen(a, b):
|
||||
"""Corridor size penalty (§B1.5 intent: size defines a BUDGET CORRIDOR; boundary class decides
|
||||
WITHIN it). Flat (0) inside [lo_band, hi_band]; steep outside so chunks stay near target size."""
|
||||
o = cum_out[b] - cum_out[a]
|
||||
if o > hard_max_out:
|
||||
return 100000 + int(40 * (o - hard_max_out)) # hard ceiling — never exceed
|
||||
if o > hi_band:
|
||||
return int(30 * (o - hi_band)) # over corridor
|
||||
if o < lo_band:
|
||||
return int(30 * (lo_band - o)) # under corridor
|
||||
return 0 # inside corridor: size is free, boundary decides
|
||||
|
||||
INF = float("inf")
|
||||
# dp[b] = min cost to segment units[0:b]; back[b] = previous cut index a
|
||||
dp = [INF] * (n + 1)
|
||||
back = [0] * (n + 1)
|
||||
dp[0] = 0
|
||||
for b in range(1, n + 1):
|
||||
# try all chunk starts a < b (bounded to a window where size is plausible)
|
||||
a = b - 1
|
||||
while a >= 0:
|
||||
o = cum_out[b] - cum_out[a]
|
||||
if o > hard_max_out * 1.6 and a < b - 1:
|
||||
break # window: chunks far over hardMax are never optimal
|
||||
cost = dp[a] + w_size * size_pen(a, b)
|
||||
if b < n: # boundary penalty on the RIGHT cut (gap b-1)
|
||||
cost += w_bound * _CLASS_PEN[gap_class[b - 1]] + w_dialog * gap_dialog[b - 1]
|
||||
if cost < dp[b]:
|
||||
dp[b] = cost
|
||||
back[b] = a
|
||||
a -= 1
|
||||
# reconstruct
|
||||
cuts = []
|
||||
b = n
|
||||
while b > 0:
|
||||
a = back[b]
|
||||
cuts.append((a, b))
|
||||
b = a
|
||||
cuts.reverse()
|
||||
chunks = []
|
||||
cut_offsets = []
|
||||
for (a, b) in cuts:
|
||||
txt = "".join(units[k][0] for k in range(a, b))
|
||||
rcls = gap_class[b - 1] if b < n else "B0"
|
||||
chunks.append((txt, est_out(txt), rcls))
|
||||
if a > 0:
|
||||
cut_offsets.append(cum_rune[a])
|
||||
return ChunkPlan(chunks=chunks, cut_offsets=cut_offsets)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if "--selftest" in sys.argv:
|
||||
text = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime.txt").read_text(encoding="utf-8")
|
||||
units = units_from_text(text)
|
||||
tiled = "".join(u[0] for u in units)
|
||||
print("units:", len(units), "tiling lossless:", tiled == text)
|
||||
assert tiled == text, "UNIT TILING NOT LOSSLESS"
|
||||
total_out = est_out(text)
|
||||
print(f"S2' total est_out={total_out:.0f} est_tokens={est_tokens(text)}")
|
||||
# size sweep: how many logical chunks at a few targets
|
||||
for tgt in (1500, 1797, 2200):
|
||||
plan = logical_chunks(text, target_out=tgt, hard_max_out=tgt * 1.25)
|
||||
outs = [c[1] for c in plan.chunks]
|
||||
classes = [c[2] for c in plan.chunks[:-1]] # right-boundary classes (excl last)
|
||||
from collections import Counter
|
||||
print(f" target_out={tgt}: {len(plan.chunks)} chunks "
|
||||
f"mean_out={sum(outs)/len(outs):.0f} min={min(outs):.0f} max={max(outs):.0f} "
|
||||
f"cut-classes={dict(Counter(classes))}")
|
||||
print("chunker selftest OK")
|
||||
100
eval/exp15/enrich_ana.py
Normal file
100
eval/exp15/enrich_ana.py
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — kimi enrichment of T-ana/T-ell straddles AT flat-A0-30 boundaries (guard 1: kimi ONLY as
|
||||
targeted enrichment on the boundaries, deterministic-first). For each greedy boundary we give kimi the
|
||||
TAIL of chunk_i and the HEAD of chunk_{i+1} and ask ONLY: is there an anaphora (esp. zh zero-subject) or
|
||||
ellipsis in the HEAD whose antecedent/elided element is in the TAIL (i.e. it straddles THIS cut)? Batched
|
||||
(several boundaries per call) to bound kimi cost. Family-isolated miner (Moonshot), temp 1, 24k + retry.
|
||||
Run: python enrich_ana.py --run
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import exp15_llm as L
|
||||
from chunker import greedy_chunks, split_source_sentences
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
FLAT = SD / "S2prime_big_flat.txt"
|
||||
OUT = SD / "ana_enrichment.json"
|
||||
LEDGER = SD / "enrich_costs.jsonl"
|
||||
MINER = "kimi-k2.6"
|
||||
BATCH = 6 # boundaries per kimi call
|
||||
TAIL_N, HEAD_N = 3, 3
|
||||
|
||||
SYS = (
|
||||
"Ты — эксперт по китайско-русскому переводу и связности. Тебе дают несколько ГРАНИЦ нарезки текста "
|
||||
"(ранобэ 蛊真人): для каждой — ХВОСТ предыдущего чанка и ГОЛОВА следующего. Для КАЖДОЙ границы найди, "
|
||||
"есть ли в ГОЛОВЕ анафора (особенно китайское НУЛЕВОЕ подлежащее — предложение без явного субъекта, "
|
||||
"субъект тянется из ХВОСТА) ИЛИ эллипсис, чей антецедент/опущенный элемент — в ХВОСТЕ (т.е. зависимость "
|
||||
"ПЕРЕСЕКАЕТ именно эту границу). Если да — верни объект; если нет — пропусти границу.\n"
|
||||
'Формат (JSON-массив): [{"boundary_idx":ЦЕЛОЕ,"type":"T-ana|T-ell",'
|
||||
'"antecedent_span":"цитата из ХВОСТА","dependent_span":"цитата из ГОЛОВЫ",'
|
||||
'"phenomenon":"какая зависимость и почему разрез её ломает",'
|
||||
'"expected_ru":"что должен сделать верный перевод (референт/род/восстановленное сказуемое)"}]\n'
|
||||
"Только реальные пересекающие границу случаи; цитаты дословные. Без текста вне JSON."
|
||||
)
|
||||
|
||||
|
||||
def boundaries():
|
||||
text = FLAT.read_text(encoding="utf-8")
|
||||
ch = greedy_chunks(text)
|
||||
out = []
|
||||
for i in range(len(ch) - 1):
|
||||
tail = "".join(split_source_sentences(ch[i])[-TAIL_N:]).strip()
|
||||
head = "".join(split_source_sentences(ch[i + 1])[:HEAD_N]).strip()
|
||||
out.append({"boundary_idx": i, "tail": tail, "head": head})
|
||||
return out
|
||||
|
||||
|
||||
def parse_array(t):
|
||||
m = re.search(r"\[.*\]", t or "", re.S)
|
||||
if not m:
|
||||
return None
|
||||
try:
|
||||
return json.loads(m.group(0))
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--run", action="store_true"); a = ap.parse_args()
|
||||
b = boundaries()
|
||||
print(f"{len(b)} boundaries, batch {BATCH} -> {(len(b)+BATCH-1)//BATCH} kimi calls")
|
||||
if not a.run:
|
||||
return
|
||||
sp = L.Spender(str(LEDGER), per_call_cap=0.28, hard_cap=2.5)
|
||||
found = []
|
||||
NUDGE = {"role": "user", "content": "Не рассуждай долго. Сразу выведи ТОЛЬКО компактный валидный JSON-массив, без markdown-ограды."}
|
||||
for k in range(0, len(b), BATCH):
|
||||
batch = b[k:k + BATCH]
|
||||
user = "\n\n".join(f"ГРАНИЦА {x['boundary_idx']}:\n[ХВОСТ]: {x['tail']}\n[ГОЛОВА]: {x['head']}" for x in batch)
|
||||
msgs = [{"role": "system", "content": SYS}, {"role": "user", "content": user}]
|
||||
in_est = (len(SYS) + len(user)) // 2
|
||||
ok, pred, why = sp.gate(MINER, in_est, L.MODELS[MINER]["max_tokens"])
|
||||
print(f"[gate] batch {k//BATCH} (bnd {batch[0]['boundary_idx']}-{batch[-1]['boundary_idx']}) pred=${pred:.4f} {'OK' if ok else why}")
|
||||
if not ok:
|
||||
continue
|
||||
r = L.call(MINER, msgs, temp=1.0, max_tokens=24000, timeout=600)
|
||||
sp.record({"batch": k // BATCH, "usage": r.get("usage"), "cost": r.get("cost", 0.0), "err": r.get("err"), "latency_s": r.get("latency_s")})
|
||||
cands = parse_array(r.get("text", ""))
|
||||
if cands is None:
|
||||
r2 = L.call(MINER, msgs + [NUDGE], temp=1.0, max_tokens=24000, timeout=600)
|
||||
sp.record({"batch": k // BATCH, "retry": True, "usage": r2.get("usage"), "cost": r2.get("cost", 0.0), "err": r2.get("err")})
|
||||
cands = parse_array(r2.get("text", "")) if not r2.get("err") else None
|
||||
cands = cands or []
|
||||
found.extend(cands)
|
||||
with open(str(OUT) + ".partial.jsonl", "a", encoding="utf-8") as pf:
|
||||
pf.write(json.dumps({"batch": k // BATCH, "n": len(cands), "cands": cands}, ensure_ascii=False) + "\n")
|
||||
print(f" batch {k//BATCH}: {len(cands)} straddles cost=${r.get('cost',0):.4f}")
|
||||
from collections import Counter
|
||||
by = Counter(c.get("type") for c in found)
|
||||
OUT.write_text(json.dumps({"n": len(found), "by_type": dict(by), "straddles": found}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\nenriched straddles: {dict(by)} (total {len(found)}) spend ${sp.total:.4f} -> {OUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
215
eval/exp15/exp15_llm.py
Normal file
215
eval/exp15/exp15_llm.py
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — shared LLM call / cost / per-call-gate / ledger layer for paid rigs (judges, miner, arms).
|
||||
|
||||
Uses the openai SDK (proven in wirecheck.py) with the frozen live prices (freeze_facts.json) — NOT
|
||||
exp14_common.PRICES (a stale 2026-07-10 mirror). Cost accounting is per-provider and VERIFIED against
|
||||
the wire-check usage:
|
||||
grok-4.3 : total = prompt + completion + reasoning => output billed = completion + reasoning (ADDITIVE)
|
||||
kimi-k2.6: verbose reasoning billed as output => output billed = completion + reasoning (ADDITIVE)
|
||||
mistral : no reasoning surcharge => output billed = completion
|
||||
glm-5 : output billed = completion; cached prompt tokens billed at cache_hit rate
|
||||
deepseek : output billed = completion (thinking folded / reported 0 for flash)
|
||||
Tiered (grok/gemini) at 200k prompt tokens. Every paid call persists raw usage + cost (D30.10).
|
||||
Per-call PREDICTED-cost gate BEFORE the call (worst-case max_tokens as output). Errors are returned,
|
||||
not raised (honest accounting: record every attempt incl. failures — D37 lesson).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from openai import OpenAI
|
||||
from dotenv import load_dotenv
|
||||
|
||||
EVAL = Path(__file__).resolve().parent.parent
|
||||
load_dotenv(EVAL / ".env")
|
||||
FACTS = json.load(open("/home/ubuntu/books/gu-zhenren/exp15/freeze_facts.json"))
|
||||
PRICES = FACTS["prices"]
|
||||
|
||||
# model -> provider wiring (base_url, env key, default temp, max_tokens floor, static extra_body,
|
||||
# whether reasoning tokens are billed additively as output)
|
||||
MODELS = {
|
||||
"deepseek-v4-flash": dict(base="https://api.deepseek.com/v1", env="DEEPSEEK_API_KEY",
|
||||
temp=0.3, max_tokens=8000, extra={}, reasoning_additive=False),
|
||||
"deepseek-v4-pro": dict(base="https://api.deepseek.com/v1", env="DEEPSEEK_API_KEY",
|
||||
temp=0.3, max_tokens=8000, extra={}, reasoning_additive=False),
|
||||
"glm-5": dict(base="https://api.z.ai/api/paas/v4", env="ZAI_API_KEY",
|
||||
temp=0.4, max_tokens=8000, extra={"thinking": {"type": "disabled"}},
|
||||
reasoning_additive=False),
|
||||
"grok-4.3": dict(base="https://api.x.ai/v1", env="XAI_API_KEY",
|
||||
temp=0.0, max_tokens=8000, extra={}, reasoning_additive=True), # reasoning ON (default)
|
||||
"mistral-large-latest": dict(base="https://api.mistral.ai/v1", env="MISTRAL_API_KEY",
|
||||
temp=0.0, max_tokens=8000, extra={"safe_prompt": False},
|
||||
reasoning_additive=False),
|
||||
"kimi-k2.6": dict(base="https://api.moonshot.ai/v1", env="KIMI_API_KEY",
|
||||
temp=1.0, max_tokens=16000, extra={}, reasoning_additive=True), # temp MUST be 1
|
||||
}
|
||||
|
||||
|
||||
def _price(model, field, prompt_tokens=0):
|
||||
v = PRICES[model][field]
|
||||
if isinstance(v, list):
|
||||
tier_at = PRICES[model].get("tier_at", 200000)
|
||||
return v[1] if prompt_tokens >= tier_at else v[0]
|
||||
return v
|
||||
|
||||
|
||||
def _reasoning_tokens(usage_obj) -> int:
|
||||
d = usage_obj if isinstance(usage_obj, dict) else getattr(usage_obj, "__dict__", {})
|
||||
if d.get("reasoning_tokens"):
|
||||
return d["reasoning_tokens"]
|
||||
ctd = d.get("completion_tokens_details")
|
||||
if ctd is not None:
|
||||
ctd = ctd if isinstance(ctd, dict) else getattr(ctd, "__dict__", {})
|
||||
return ctd.get("reasoning_tokens", 0) or 0
|
||||
# xAI: total = prompt + completion + reasoning -> infer if fields present
|
||||
return 0
|
||||
|
||||
|
||||
def _cached_tokens(usage_obj) -> int:
|
||||
d = usage_obj if isinstance(usage_obj, dict) else getattr(usage_obj, "__dict__", {})
|
||||
# DeepSeek surfaces cache hits at top-level (prompt_cache_hit_tokens); GLM/OpenAI/Gemini nest it.
|
||||
if d.get("prompt_cache_hit_tokens"):
|
||||
return d["prompt_cache_hit_tokens"]
|
||||
ptd = d.get("prompt_tokens_details")
|
||||
if ptd is not None:
|
||||
ptd = ptd if isinstance(ptd, dict) else getattr(ptd, "__dict__", {})
|
||||
return ptd.get("cached_tokens", 0) or 0
|
||||
return 0
|
||||
|
||||
|
||||
def cost(model, usage: dict) -> float:
|
||||
pt = usage.get("prompt_tokens", 0)
|
||||
ct = usage.get("completion_tokens", 0)
|
||||
rt = usage.get("reasoning_tokens", 0)
|
||||
cached = usage.get("cached_tokens", 0)
|
||||
pin = _price(model, "in", pt)
|
||||
pout = _price(model, "out", pt)
|
||||
phit = _price(model, "cache_hit", pt) if "cache_hit" in PRICES[model] else pin
|
||||
out_tok = ct + (rt if MODELS[model]["reasoning_additive"] else 0)
|
||||
in_cost = (pt - cached) * pin + cached * phit
|
||||
return (in_cost + out_tok * pout) / 1e6
|
||||
|
||||
|
||||
# reasoning tokens are NOT capped by max_tokens for these models -> the gate must budget them too,
|
||||
# else predict_cost under-estimates and a call can overshoot per_call_cap (kimi ~12k reasoning/para).
|
||||
REASONING_CEIL = {"kimi-k2.6": 14000, "grok-4.3": 4000}
|
||||
|
||||
|
||||
def predict_cost(model, in_tok, max_out) -> float:
|
||||
"""Worst-case gate estimate: full max_tokens as billed output PLUS an uncapped-reasoning allowance
|
||||
for additive-reasoning models (kimi/grok), whose reasoning tokens are billed but not bounded by
|
||||
max_tokens. Without this the per-call/hard gate under-predicts (seen: kimi $0.124 vs $0.066)."""
|
||||
pin = _price(model, "in", in_tok)
|
||||
pout = _price(model, "out", in_tok)
|
||||
out_est = max_out + (REASONING_CEIL.get(model, 0) if MODELS[model]["reasoning_additive"] else 0)
|
||||
return (in_tok * pin + out_est * pout) / 1e6
|
||||
|
||||
|
||||
class Spender:
|
||||
"""Per-call predicted-cost gate + append-only jsonl ledger + running total (D30.10)."""
|
||||
|
||||
def __init__(self, ledger_path: str, per_call_cap: float, hard_cap: float = None, seed_total: float = 0.0):
|
||||
# seed_total = spend already booked in OTHER ledgers, so hard_cap can be a GLOBAL ceiling even
|
||||
# though each block writes its own per-block ledger (review I8: per-block caps summed to 4x budget).
|
||||
self.path = Path(ledger_path)
|
||||
self.per_call_cap = per_call_cap
|
||||
self.hard_cap = hard_cap
|
||||
self.seed_total = seed_total
|
||||
self.total = seed_total
|
||||
if self.path.exists():
|
||||
for line in self.path.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
self.total += json.loads(line).get("cost", 0.0)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def gate(self, model, in_tok, max_out):
|
||||
pred = predict_cost(model, in_tok, max_out)
|
||||
if pred > self.per_call_cap:
|
||||
return False, pred, f"per-call gate: pred ${pred:.4f} > cap ${self.per_call_cap}"
|
||||
if self.hard_cap is not None and self.total + pred > self.hard_cap:
|
||||
return False, pred, f"hard-cap gate: total ${self.total:.4f}+${pred:.4f} > ${self.hard_cap}"
|
||||
return True, pred, "ok"
|
||||
|
||||
def record(self, rec: dict):
|
||||
c = rec.get("cost", 0.0)
|
||||
self.total += c
|
||||
with self.path.open("a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
# proactive per-model min interval between calls (avoid 429 storms; mistral rate-limits aggressively).
|
||||
# FLOOR full-chunk run measured mistral at 48% retry-fails @1.3s (max lat 64.7s) vs grok 0% @0.4s -> widen
|
||||
# mistral spacing so its per-cell burst stays under the rate limit (fewer backoffs => cleaner AND faster).
|
||||
MIN_INTERVAL = {"mistral-large-latest": 2.6, "grok-4.3": 0.4, "kimi-k2.6": 0.0}
|
||||
_LAST_CALL = {}
|
||||
|
||||
|
||||
def call(model, messages, temp=None, max_tokens=None, extra_body=None, timeout=240, retries=4):
|
||||
"""Return {text, reasoning, usage, cost, model_returned, finish, err, latency_s}. Never raises.
|
||||
Retries transient errors (429 rate-limit, 5xx, timeouts) with exponential backoff (5/15/35/60s)."""
|
||||
m = MODELS[model]
|
||||
key = os.environ.get(m["env"])
|
||||
if not key:
|
||||
return {"err": f"no key {m['env']}", "cost": 0.0, "usage": {}}
|
||||
mi = MIN_INTERVAL.get(model, 0.0)
|
||||
if mi:
|
||||
dt = time.time() - _LAST_CALL.get(model, 0.0)
|
||||
if dt < mi:
|
||||
time.sleep(mi - dt)
|
||||
_LAST_CALL[model] = time.time()
|
||||
cli = OpenAI(base_url=m["base"], api_key=key, timeout=timeout)
|
||||
body_extra = dict(m["extra"])
|
||||
if extra_body:
|
||||
body_extra.update(extra_body)
|
||||
kw = dict(model=model, messages=messages,
|
||||
temperature=m["temp"] if temp is None else temp,
|
||||
max_tokens=max_tokens or m["max_tokens"])
|
||||
if body_extra:
|
||||
kw["extra_body"] = body_extra
|
||||
t0 = time.time()
|
||||
backoff = [5, 15, 35, 60]
|
||||
last_err = None
|
||||
retry_fails = 0 # transient attempts that were retried (may have billed server-side; review I8)
|
||||
for attempt in range(retries + 1):
|
||||
try:
|
||||
r = cli.chat.completions.create(**kw)
|
||||
break
|
||||
except Exception as e: # noqa: BLE001
|
||||
last_err = f"{type(e).__name__}: {str(e)[:200]}"
|
||||
transient = any(s in last_err.lower() for s in ("ratelimit", "429", "rate limit", "timeout",
|
||||
"timed out", "500", "502", "503", "529", "overloaded"))
|
||||
if attempt < retries and transient:
|
||||
retry_fails += 1
|
||||
time.sleep(backoff[min(attempt, len(backoff) - 1)])
|
||||
continue
|
||||
return {"err": last_err, "cost": 0.0, "usage": {}, "latency_s": round(time.time() - t0, 1),
|
||||
"retry_fails": retry_fails}
|
||||
u = r.usage
|
||||
usage = {"prompt_tokens": getattr(u, "prompt_tokens", 0),
|
||||
"completion_tokens": getattr(u, "completion_tokens", 0),
|
||||
"total_tokens": getattr(u, "total_tokens", 0),
|
||||
"reasoning_tokens": _reasoning_tokens(u),
|
||||
"cached_tokens": _cached_tokens(u)}
|
||||
# xAI: reasoning often only inferable from total-prompt-completion
|
||||
if usage["reasoning_tokens"] == 0 and usage["total_tokens"] > usage["prompt_tokens"] + usage["completion_tokens"]:
|
||||
usage["reasoning_tokens"] = usage["total_tokens"] - usage["prompt_tokens"] - usage["completion_tokens"]
|
||||
msg = r.choices[0].message
|
||||
text = (msg.content or "").strip()
|
||||
reasoning = (getattr(msg, "reasoning_content", None) or "").strip()
|
||||
c = cost(model, usage)
|
||||
return {"text": text, "reasoning": reasoning, "usage": usage, "cost": round(c, 6),
|
||||
"model_returned": getattr(r, "model", None), "finish": r.choices[0].finish_reason,
|
||||
"err": None, "latency_s": round(time.time() - t0, 1), "retry_fails": retry_fails}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# smoke: cost accounting sanity vs the frozen wire-check numbers
|
||||
grok_usage = {"prompt_tokens": 223, "completion_tokens": 11, "reasoning_tokens": 262, "total_tokens": 496, "cached_tokens": 0}
|
||||
print("grok additive cost (should exceed the wire-check's completion-only $0.000306):",
|
||||
round(cost("grok-4.3", grok_usage), 6))
|
||||
print(" = (223*1.25 + (11+262)*2.50)/1e6 =", round((223*1.25 + 273*2.50)/1e6, 6))
|
||||
mis_usage = {"prompt_tokens": 41, "completion_tokens": 15, "reasoning_tokens": 0, "total_tokens": 56, "cached_tokens": 0}
|
||||
print("mistral cost:", round(cost("mistral-large-latest", mis_usage), 6), "= (41*0.5+15*1.5)/1e6 =", round((41*0.5+15*1.5)/1e6, 6))
|
||||
90
eval/exp15/fertility_calib.py
Normal file
90
eval/exp15/fertility_calib.py
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — fertility calibration B1.2 ($0, offline): src chars -> ru output-token coefficients.
|
||||
|
||||
Regresses ru output tokens of the 25-chapter rerun finals on (cjk, other) source-char counts, giving
|
||||
the chunker's est_out = f_cjk*cjk + f_other*other (§B1.1/B1.2). Chunker-spec deliverable feeding the
|
||||
Q1 size-match. Offline tokenizer per §B1.2: deepseek-v4 tokenizer dir is EMPTY in repo, so we use the
|
||||
editor-family glm-4.6 tokenizer (present) and label it; Go needs only the COEFFICIENTS, not a tokenizer.
|
||||
|
||||
Run: eval/.venv/bin/python eval/exp15/fertility_calib.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
REC = Path("/home/ubuntu/books/gu-zhenren/rerun/records.json")
|
||||
TOK = Path("/home/ubuntu/projects/textmachine/eval/tokenizers/glm-4.6/tokenizer.json")
|
||||
OUT = Path("/home/ubuntu/books/gu-zhenren/exp15/fertility_calib.json")
|
||||
|
||||
|
||||
def is_cjk(ch: str) -> bool:
|
||||
o = ord(ch)
|
||||
return (0x4E00 <= o <= 0x9FFF or 0x3040 <= o <= 0x30FF or 0x3400 <= o <= 0x4DBF
|
||||
or 0xF900 <= o <= 0xFAFF or 0x20000 <= o <= 0x2A6DF)
|
||||
|
||||
|
||||
def char_classes(s: str):
|
||||
cjk = sum(1 for c in s if is_cjk(c))
|
||||
other = sum(1 for c in s if not c.isspace() and not is_cjk(c))
|
||||
return cjk, other
|
||||
|
||||
|
||||
def main():
|
||||
tok = Tokenizer.from_file(str(TOK))
|
||||
recs = json.load(open(REC))
|
||||
rows, meta = [], []
|
||||
for r in recs:
|
||||
src = r.get("source") or ""
|
||||
fin = r.get("final") or ""
|
||||
if not src.strip() or not fin.strip():
|
||||
continue
|
||||
cjk, other = char_classes(src)
|
||||
out_tok = len(tok.encode(fin).ids)
|
||||
if cjk + other < 200: # skip tiny/metadata chunks (e.g. title block)
|
||||
continue
|
||||
rows.append((cjk, other, out_tok))
|
||||
meta.append((r.get("chapter"), r.get("chunk_idx")))
|
||||
|
||||
A = np.array([[c, o] for c, o, _ in rows], dtype=float)
|
||||
y = np.array([t for _, _, t in rows], dtype=float)
|
||||
# non-negative-ish least squares, no intercept (est_out = f_cjk*cjk + f_other*other)
|
||||
coef, *_ = np.linalg.lstsq(A, y, rcond=None)
|
||||
f_cjk, f_other = float(coef[0]), float(coef[1])
|
||||
pred = A @ coef
|
||||
ss_res = float(np.sum((y - pred) ** 2))
|
||||
ss_tot = float(np.sum((y - y.mean()) ** 2))
|
||||
r2 = 1 - ss_res / ss_tot if ss_tot else float("nan")
|
||||
mape = float(np.mean(np.abs((y - pred) / y))) * 100
|
||||
|
||||
result = {
|
||||
"n_records": len(rows),
|
||||
"tokenizer": "glm-4.6 (editor family; deepseek-v4 tokenizer dir empty in repo — §B1.2 caveat)",
|
||||
"model": "est_out = f_cjk*cjk_src + f_other*other_src (no intercept)",
|
||||
"f_cjk": round(f_cjk, 4),
|
||||
"f_other": round(f_other, 4),
|
||||
"r2": round(r2, 4),
|
||||
"mape_pct": round(mape, 2),
|
||||
"placeholder_B1_1": {"cjk_char": 0.75, "other_char": 0.25},
|
||||
"note": "per-language-pair config (layer 2); recompute = loud resnapshot. Src-char est is the SECOND "
|
||||
"quantity (§B1.2); primary budget is ru-out tokens. Calibrated on 25-ch guzhenren rerun (in-pretrain).",
|
||||
}
|
||||
OUT.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
print(f"fertility calibration (n={len(rows)} chunks, glm-4.6 ru tokens):")
|
||||
print(f" est_out = {f_cjk:.4f}*cjk + {f_other:.4f}*other R2={r2:.4f} MAPE={mape:.1f}%")
|
||||
print(f" vs §B1.1 placeholder cjk=0.75/other=0.25")
|
||||
print(f" sample src cjk/other -> out (first 6):")
|
||||
for (c, o, t), m in list(zip(rows, meta))[:6]:
|
||||
print(f" ch{m[0]}#{m[1]}: cjk={c} other={o} out_tok={t} pred={f_cjk*c+f_other*o:.0f}")
|
||||
print(f"-> {OUT}")
|
||||
# self-review: coefficients must be positive and R2 reasonable
|
||||
assert f_cjk > 0, f"non-positive cjk fertility {f_cjk}"
|
||||
assert r2 > 0.5, f"poor fit R2={r2} — investigate before trusting coefficients"
|
||||
print("OK: positive fertility coefficients, R2>0.5.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
198
eval/exp15/gender_transport.py
Normal file
198
eval/exp15/gender_transport.py
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — deterministic gender transport for the memory-bank seed (owner decision #2, 16.07).
|
||||
|
||||
Mirrors the exp14b_reseed_promote provenance pattern: IDEMPOTENT + byte-stable + one-time
|
||||
pre-snapshot + provenance log + loud self-review asserts. The `gender` field ALREADY exists in
|
||||
guzhenren-seed-v2.yaml and is consumed by Go (memseed.go:35 -> memory.go:273 hash -> store;
|
||||
migrate.go:183 valid values male|female|hidden|''). So this transport does three jobs:
|
||||
|
||||
(1) AUDIT gender coverage of every person (type=name) record and EMIT an owner-signable
|
||||
disputed/missing list (`gender_signoff.yaml` template) — the T-gen trap subjects and the
|
||||
Q3a' state-arm need signed gender.
|
||||
(2) APPLY only owner-SIGNED additions from `gender_signoff.yaml` (src -> male|female|hidden),
|
||||
idempotently + byte-stably. n-a rows document "confirmed no gender field" (clan/place/concept)
|
||||
and are NOT mutated. No signoff file, or seed already complete -> NO-OP (snapshot hash preserved).
|
||||
(3) SELF-REVIEW by execution: build the eval-side scene-state slot from matched entries and assert
|
||||
gender surfaces for gendered persons (this is the Q3a' state mechanism; hidden -> gender-avoidant).
|
||||
|
||||
Vocabulary map (prompt shorthand -> DB string, migrate.go:183): m->male f->female hidden->hidden n-a->''(absent).
|
||||
Discipline: seed is OUTSIDE git; population is owner's signature (approved-invariant). If signature does
|
||||
not arrive before Q3a', the state-arm is honestly marked "state without gender", NOT imitated (prompt #2).
|
||||
|
||||
Run: eval/.venv/bin/python eval/exp15/gender_transport.py # audit + apply signed (default)
|
||||
eval/.venv/bin/python eval/exp15/gender_transport.py --audit # audit only, never mutate
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
BOOK = Path("/home/ubuntu/books/gu-zhenren")
|
||||
SEED = BOOK / "guzhenren-seed-v2.yaml"
|
||||
SIGNOFF = BOOK / "exp15" / "gender_signoff.yaml" # owner-editable; src -> gender
|
||||
BAK = BOOK / "exp15" / "guzhenren-seed-v2.pre-gender.yaml" # one-time immutable "before"
|
||||
LOG = BOOK / "exp15" / "gender_transport.md"
|
||||
DUMP = dict(allow_unicode=True, sort_keys=False, width=200) # SAME params that wrote the seed (byte-stable)
|
||||
VALID_DB = {"male", "female", "hidden"} # values that mutate a record
|
||||
VALID_SIGN = VALID_DB | {"n-a"} # n-a = documented, no field
|
||||
SHORT = {"m": "male", "f": "female", "hidden": "hidden", "n-a": "n-a",
|
||||
"male": "male", "female": "female"}
|
||||
|
||||
|
||||
def load_yaml(p: Path):
|
||||
return yaml.safe_load(p.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def persons(data):
|
||||
return [t for t in data["terms"] if t.get("type") == "name"]
|
||||
|
||||
|
||||
def load_signoff():
|
||||
"""Owner-signed src->gender. Absent => empty (pure audit / no additions)."""
|
||||
if not SIGNOFF.exists():
|
||||
return {}
|
||||
raw = load_yaml(SIGNOFF) or {}
|
||||
entries = raw.get("gender", raw) if isinstance(raw, dict) else {}
|
||||
out = {}
|
||||
for src, g in entries.items():
|
||||
g = str(g).strip()
|
||||
norm = SHORT.get(g, g)
|
||||
assert norm in VALID_SIGN, f"signoff bad value for {src!r}: {g!r} (allowed m/f/hidden/n-a)"
|
||||
out[src] = norm
|
||||
return out
|
||||
|
||||
|
||||
def apply_signed(data, signoff):
|
||||
"""Mutate in place from SIGNED table; return list of (src, old, new). Idempotent."""
|
||||
by_src = {t.get("src"): t for t in data["terms"]}
|
||||
applied, warn = [], []
|
||||
for src, g in signoff.items():
|
||||
if src not in by_src:
|
||||
warn.append(src)
|
||||
continue
|
||||
rec = by_src[src]
|
||||
if g == "n-a":
|
||||
# documented no-gender; assert the record indeed has no meaningful gender
|
||||
if rec.get("gender"):
|
||||
warn.append(f"{src}: signed n-a but record has gender={rec['gender']!r}")
|
||||
continue
|
||||
old = rec.get("gender")
|
||||
if old != g:
|
||||
rec["gender"] = g
|
||||
applied.append((src, old, g))
|
||||
return applied, warn
|
||||
|
||||
|
||||
def scene_state_slot(data, chunk, chapter=15):
|
||||
"""Q3a' state mechanism (eval-side): deterministic scene-state slot from matched seed entries,
|
||||
including gender (hidden -> gender-avoidant instruction). Substring match on Han keys."""
|
||||
lines = []
|
||||
for t in persons(data):
|
||||
src = t.get("src")
|
||||
if src and src in chunk and int(t.get("since_ch", 0)) <= chapter:
|
||||
dst = (t.get("dst") or "").strip()
|
||||
g = t.get("gender") or ""
|
||||
if g == "male":
|
||||
tag = "муж. (прош. вр. м.р.)"
|
||||
elif g == "female":
|
||||
tag = "жен. (прош. вр. ж.р.)"
|
||||
elif g == "hidden":
|
||||
tag = "пол скрыт до раскрытия -> гендерно-нейтральные конструкции"
|
||||
else:
|
||||
tag = "пол НЕ задан"
|
||||
lines.append(f"- {dst}: {tag}")
|
||||
return lines
|
||||
|
||||
|
||||
def emit_signoff_template(data, missing):
|
||||
"""Write a signoff template ONLY if none exists — pre-fill current gendered persons + disputed rows."""
|
||||
if SIGNOFF.exists():
|
||||
return False
|
||||
doc = ["# exp15 gender sign-off (owner) — src -> m | f | hidden | n-a",
|
||||
"# m/f = person; hidden = gender concealed until reveal (render gender-avoidant);",
|
||||
"# n-a = clan/place/concept (NO gender field). Edit values, then re-run gender_transport.py.",
|
||||
"# Rows already gendered in seed are shown for confirmation; DISPUTED rows need your call.",
|
||||
"gender:"]
|
||||
for t in persons(data):
|
||||
src, dst, g = t.get("src"), t.get("dst"), t.get("gender")
|
||||
if g in VALID_DB:
|
||||
doc.append(f" {src}: {g} # {dst} (already in seed; confirm)")
|
||||
for t in missing:
|
||||
doc.append(f" {t.get('src')}: DISPUTED # {t.get('dst')} — NEEDS YOUR CALL (person=m/f, else n-a)")
|
||||
SIGNOFF.write_text("\n".join(doc) + "\n", encoding="utf-8")
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
audit_only = "--audit" in sys.argv
|
||||
data0 = load_yaml(SEED)
|
||||
ppl = persons(data0)
|
||||
gendered = [t for t in ppl if t.get("gender") in VALID_DB]
|
||||
missing = [t for t in ppl if t.get("gender") not in VALID_DB]
|
||||
|
||||
signoff = load_signoff()
|
||||
template_written = emit_signoff_template(data0, missing)
|
||||
|
||||
# one-time immutable "before"
|
||||
if not audit_only and not BAK.exists():
|
||||
BAK.write_text(SEED.read_text(encoding="utf-8"), encoding="utf-8")
|
||||
|
||||
data = load_yaml(SEED)
|
||||
applied, warn = ([], [])
|
||||
if not audit_only and signoff:
|
||||
applied, warn = apply_signed(data, signoff)
|
||||
new_bytes = yaml.dump(data, **DUMP)
|
||||
if applied:
|
||||
SEED.write_text(new_bytes, encoding="utf-8")
|
||||
|
||||
# ---- self-review: scene-state slot surfaces gender ----
|
||||
demo_chunk = "".join(t["src"] for t in gendered[:3]) + "在村中相遇。"
|
||||
slot = scene_state_slot(data, demo_chunk)
|
||||
|
||||
# ---- provenance log ----
|
||||
lg = ["# exp15 — gender transport (owner decision #2, 2026-07-16)", "",
|
||||
f"Seed: `{SEED.name}` (OUTSIDE git). Pattern: exp14b_reseed_promote (idempotent, byte-stable, pre-snapshot+log).",
|
||||
f"Gender already consumed by Go: memseed.go:35 -> memory.go:273 (hashed) -> store; migrate.go:183 = male|female|hidden|''.",
|
||||
"", f"Person (type=name) records: {len(ppl)}; gendered {len(gendered)}; needing signature {len(missing)}.",
|
||||
f"Signed additions applied this run: {len(applied)} (0 = seed already complete / no signoff -> snapshot hash UNCHANGED).",
|
||||
""]
|
||||
lg += ["## Gendered persons (already signed via approved status)", "| src | dst | gender |", "|---|---|---|"]
|
||||
for t in gendered:
|
||||
lg.append(f"| {t.get('src')} | {t.get('dst')} | {t.get('gender')} |")
|
||||
lg += ["", "## DISPUTED / missing gender — need owner signature", "| src | dst | note |", "|---|---|---|"]
|
||||
for t in missing:
|
||||
lg.append(f"| {t.get('src')} | {t.get('dst')} | {(t.get('note') or '')[:80]} |")
|
||||
if applied:
|
||||
lg += ["", "## Applied signed additions this run", "| src | old | new |", "|---|---|---|"]
|
||||
lg += [f"| {s} | {o} | {n} |" for s, o, n in applied]
|
||||
if warn:
|
||||
lg += ["", "⚠ warnings: " + "; ".join(warn)]
|
||||
lg += ["", "## Self-review — Q3a' scene-state slot (eval-side, gender surfaces)",
|
||||
f"synthetic chunk (my construct): `{demo_chunk}`", "```", *(slot or ["(empty)"]), "```"]
|
||||
if not audit_only:
|
||||
LOG.write_text("\n".join(lg), encoding="utf-8")
|
||||
|
||||
# ---- console ----
|
||||
print(f"{'[AUDIT] ' if audit_only else ''}persons={len(ppl)} gendered={len(gendered)} need-signature={len(missing)} applied={len(applied)}")
|
||||
print("gendered:", ", ".join(f"{t.get('src')}={t.get('gender')}" for t in gendered))
|
||||
print("NEEDS SIGNATURE:", ", ".join(f"{t.get('src')}({t.get('dst')})" for t in missing) or "(none)")
|
||||
if template_written:
|
||||
print(f"-> wrote sign-off TEMPLATE: {SIGNOFF} (owner edits DISPUTED rows, then re-run)")
|
||||
print("-- scene-state slot (self-review) --")
|
||||
for x in (slot or ["(empty)"]):
|
||||
print(" " + x)
|
||||
|
||||
# ---- loud asserts (self-review by execution) ----
|
||||
assert all(t.get("gender") in VALID_DB for t in gendered), "gendered set contains a bad value"
|
||||
# every gendered person surfaces its tag in the slot when present in chunk:
|
||||
for t in gendered[:3]:
|
||||
assert any(t.get("dst") in line for line in slot), f"state slot dropped {t.get('dst')}"
|
||||
assert not warn or audit_only, f"apply warnings: {warn}"
|
||||
print("\nOK: gender field consumed by Go, transport idempotent/byte-stable, state slot surfaces gender.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
300
eval/exp15/judge_run.py
Normal file
300
eval/exp15/judge_run.py
Normal file
|
|
@ -0,0 +1,300 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — judge driver for the carrying block (Q1b oracle-vs-flat + Q3a carryover + Q0 + FLOOR), §D3.
|
||||
|
||||
Maps each final_traps.json trap to the chunk whose SOURCE contains its dependent_span, in each of:
|
||||
flat-A0-30 (blind baseline) oracle-30 (seam-aligned) arm modes q3a-src/draft/final
|
||||
then gives the pairwise span judge (judges.py: both-orders, 6->10, LOO, catastrophe, per-vote persist)
|
||||
the trap + a DEPENDENT-CENTRED SYMMETRIC WINDOW of each side's translation.
|
||||
|
||||
WINDOW POLICY (orchestrator CRITICAL fix, replaces the old HEAD_CHARS=1100 head-excerpt):
|
||||
The old head-excerpt showed the judge only the first 1100 chars of each final. In the FLAT arm the
|
||||
straddle-dependent lands at frac ~0 (chunk head) -> always visible. In the ORACLE arm the SAME
|
||||
dependent sits mid-chunk (frac 0.09-1.0) -> its Russian rendering fell PAST char 1100 in 13/18
|
||||
T-ana cells, so the judge scored "absent"=wrong/catastrophe by construction. That artifact, not the
|
||||
boundary, produced the -0.564 headline. Fix: a fixed-length window centred on the dependent's
|
||||
PROPORTIONAL position in each side's target, SAME length L for every arm (size-symmetric). Both
|
||||
sides now always contain the local phenomenon (anchor+dependent sit within a few hundred source
|
||||
chars of each other -> inside one window). The floor block (A0 vs A0') validates the policy: if the
|
||||
window systematically hid phenomena the floor would be large/asymmetric.
|
||||
|
||||
Q1b: a0=flat, arm=oracle (does cutting AT the scene boundary help vs a blind mid-scene cut?)
|
||||
Q3a: a0=flat, arm=mode (does carryover fix the cross-boundary dependency? per mode)
|
||||
Q0 : a0=do-nothing (draft), arm=A0 (editor final) (TOST equivalence — APE lesson)
|
||||
FLOOR: a0=flat-A0, arm=flat-A0' (independent regen, same config) — the judge+generation NOISE floor;
|
||||
effects below it are not interpretable (freeze §1.5). Same window policy as every block.
|
||||
|
||||
Oracle mapping is REQUIRED only for Q1b. Flat mapping is mandatory for all blocks. (Old code required
|
||||
BOTH -> T-ana-33, whose oracle chunk 12 was cjk_artifact-skipped, was silently dropped from Q3a/Q0/FLOOR
|
||||
too. Now flat-only blocks keep it; Q1b documents its exclusion.)
|
||||
|
||||
Judge SUBSET (budget scope, guard 5): ALL T-ana (primary) + ALL T-ell + a capped T-ent CONTROL sample.
|
||||
--project : $0 cell/token/cost estimate (guard-5 projection; NO api calls)
|
||||
--run Q1b|Q3a|Q0|FLOOR : execute that block | --tent-control N : cap the T-ent control cells
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import exp15_llm as L
|
||||
from judges import judge_cell, JUDGES
|
||||
from chunker import greedy_chunks
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
AH = SD / "anchors"
|
||||
ARMS = SD / "arms"
|
||||
WIN_CHARS = 1500 # dependent-centred symmetric window fed to the judge (same L for both arms)
|
||||
GLOBAL_CEILING = 14.5 # hard global spend ceiling across ALL blocks/ledgers (review I8; $15 freeze - margin)
|
||||
# Orchestrator's AUTHORITATIVE pre-session-2 spend (2541 billing lines, 0 discrepancies). Used as the seed
|
||||
# floor because it includes the tmctl anchor-generation (flat/oracle/a0prime .db) that ran through the Go
|
||||
# backend, NOT the Python Spender ledgers — so a raw ledger-sum would under-count the true global by ~$2.
|
||||
ALREADY_SPENT = 9.82
|
||||
NEW_LEDGERS = ("judge_costs_FLOOR.jsonl", "judge_costs_Q1b.jsonl", "judge_costs_Q4a.jsonl")
|
||||
|
||||
|
||||
def global_spent_excluding(block):
|
||||
"""ALREADY_SPENT floor + this session's NEW re-run ledgers (except the current block's own, which its
|
||||
own Spender loads). Enforces a true global ceiling via seed_total (review I8)."""
|
||||
tot = ALREADY_SPENT
|
||||
for name in NEW_LEDGERS:
|
||||
if name == f"judge_costs_{block}.jsonl":
|
||||
continue
|
||||
p = SD / name
|
||||
if p.exists():
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
tot += json.loads(line).get("cost", 0.0)
|
||||
except Exception:
|
||||
pass
|
||||
return tot
|
||||
|
||||
|
||||
def _db_stage(db, stage):
|
||||
con = sqlite3.connect(db)
|
||||
out = {}
|
||||
for ck, txt in con.execute("SELECT chunk_idx, response_text FROM checkpoints WHERE stage=? "
|
||||
"ORDER BY chunk_idx, attempt", (stage,)):
|
||||
out[ck] = txt
|
||||
con.close()
|
||||
return out
|
||||
|
||||
|
||||
def db_finals(db):
|
||||
return _db_stage(db, "edit")
|
||||
|
||||
|
||||
def db_drafts(db):
|
||||
return _db_stage(db, "draft")
|
||||
|
||||
|
||||
def find_off(chunks, span):
|
||||
"""(chunk_idx, char_offset) of the first chunk whose source contains span's 20-char head."""
|
||||
probe = (span or "").strip()[:20]
|
||||
for i, c in enumerate(chunks):
|
||||
if probe and probe in c:
|
||||
return i, c.find(probe)
|
||||
return None, None
|
||||
|
||||
|
||||
def locate_flat(trap, flat_src):
|
||||
"""Prefer the trap's authoritative dep_chunk (verified to contain the dependent); else scan."""
|
||||
d = (trap["dependent_span"] or "").strip()[:20]
|
||||
fi = trap.get("dep_chunk")
|
||||
if fi is not None and 0 <= fi < len(flat_src) and d and d in flat_src[fi]:
|
||||
return fi, flat_src[fi].find(d)
|
||||
return find_off(flat_src, trap["dependent_span"])
|
||||
|
||||
|
||||
def frac(chunk_src, off):
|
||||
n = len(chunk_src or "")
|
||||
return (off / n) if (n and off is not None and off >= 0) else 0.0
|
||||
|
||||
|
||||
def window(text, fr, L=WIN_CHARS):
|
||||
"""Fixed-length (L) window of `text` centred on proportional position `fr`, clamped-and-extended
|
||||
so it is always exactly L chars when text is longer than L (size-symmetric across arms)."""
|
||||
text = text or ""
|
||||
if len(text) <= L:
|
||||
return text
|
||||
center = int(round(fr * len(text)))
|
||||
lo, hi = center - L // 2, center - L // 2 + L
|
||||
if lo < 0:
|
||||
lo, hi = 0, L
|
||||
if hi > len(text):
|
||||
hi, lo = len(text), len(text) - L
|
||||
return text[lo:hi]
|
||||
|
||||
|
||||
def load():
|
||||
flat_src = greedy_chunks((SD / "S2prime_big_flat.txt").read_text(encoding="utf-8"))
|
||||
oracle_src = greedy_chunks((SD / "S2prime_big_blanklines.txt").read_text(encoding="utf-8"))
|
||||
flat_fin = db_finals(AH / "flat_a0_30.db")
|
||||
oracle_fin = db_finals(AH / "oracle_30.db")
|
||||
traps = json.load(open(SD / "final_traps.json"))["traps"]
|
||||
return flat_src, oracle_src, flat_fin, oracle_fin, traps
|
||||
|
||||
|
||||
def arm_final(mode, chunk_idx):
|
||||
p = ARMS / mode / f"{chunk_idx:02d}.final.txt"
|
||||
return p.read_text(encoding="utf-8") if p.exists() else None
|
||||
|
||||
|
||||
def locate_oracle(trap, oracle_src):
|
||||
"""Select the oracle chunk holding the phenomenon. Prefer a chunk containing BOTH the anchor and the
|
||||
dependent (seam-aligned -> co-located for T-ana/T-ell). Else accept a dependent-only chunk ONLY when
|
||||
the dependent probe is distinctive (>=8 chars) AND unique — short/ambiguous probes are skipped rather
|
||||
than judged on the wrong passage (review I3). Returns (chunk_idx, dep_offset) or (None, None)."""
|
||||
d = (trap["dependent_span"] or "").strip()[:20]
|
||||
a = (trap["anchor_span"] or "").strip()[:20]
|
||||
for i, c in enumerate(oracle_src):
|
||||
if d and a and d in c and a in c:
|
||||
return i, c.find(d)
|
||||
hits = [i for i, c in enumerate(oracle_src) if d and d in c]
|
||||
if len(d) >= 8 and len(hits) == 1:
|
||||
return hits[0], oracle_src[hits[0]].find(d)
|
||||
return None, None
|
||||
|
||||
|
||||
def excerpt(final, fr, full):
|
||||
"""Full final (no windowing) when full=True; else the dependent-centred window."""
|
||||
return final if full else window(final, fr)
|
||||
|
||||
|
||||
def map_trap(trap, flat_src, oracle_src, flat_fin, oracle_fin, full=False):
|
||||
"""Flat mapping is mandatory (returns None if absent). Oracle mapping is optional (None if absent
|
||||
-> Q1b skips, flat-only blocks proceed). full=True feeds whole finals (no windowing)."""
|
||||
fi, fpos = locate_flat(trap, flat_src)
|
||||
if fi is None or fi not in flat_fin:
|
||||
return None
|
||||
ffr = frac(flat_src[fi], fpos)
|
||||
jt = {"id": trap["id"], "type": trap["type"],
|
||||
"src_zone": trap["anchor_span"] + " … " + trap["dependent_span"],
|
||||
"phenomenon": trap["phenomenon"], "expected": trap["expected_ru"]}
|
||||
orac = None
|
||||
oi, opos = locate_oracle(trap, oracle_src)
|
||||
if oi is not None and oi in oracle_fin:
|
||||
ofr = frac(oracle_src[oi], opos)
|
||||
orac = {"chunk": oi, "frac": round(ofr, 3), "win": excerpt(oracle_fin[oi], ofr, full)}
|
||||
return {"jt": jt, "flat_chunk": fi, "flat_frac": round(ffr, 3),
|
||||
"flat_win": excerpt(flat_fin[fi], ffr, full), "oracle": orac}
|
||||
|
||||
|
||||
def select_traps(traps, tent_control, primary_only=False):
|
||||
prim = [t for t in traps if t["type"] in ("T-ana", "T-ell")]
|
||||
if primary_only:
|
||||
return prim
|
||||
tent = [t for t in traps if t["type"] == "T-ent"][:tent_control]
|
||||
return prim + tent
|
||||
|
||||
|
||||
def project(traps, tent_control):
|
||||
flat_src, oracle_src, flat_fin, oracle_fin, _ = load()
|
||||
sub = select_traps(traps, tent_control)
|
||||
mapped = [t for t in sub if map_trap(t, flat_src, oracle_src, flat_fin, oracle_fin)]
|
||||
n = len(mapped)
|
||||
q1b_ok = sum(1 for t in mapped
|
||||
if (map_trap(t, flat_src, oracle_src, flat_fin, oracle_fin) or {}).get("oracle"))
|
||||
in_tok = (600 + 400 + 2 * WIN_CHARS // 2) # sys + trap + 2 windows (~2 chars/token)
|
||||
votes = 7.4 # 6 base + 0.35*4 escalation avg, per judge x2
|
||||
g = (in_tok * L._price("grok-4.3", "in") + 900 * L._price("grok-4.3", "out")) / 1e6
|
||||
m = (in_tok * L._price("mistral-large-latest", "in") + 200 * L._price("mistral-large-latest", "out")) / 1e6
|
||||
cc = votes * (g + m)
|
||||
q3a_cells = sum(1 for t in mapped if t["type"] in ("T-ana", "T-ell")) * 3 + \
|
||||
sum(1 for t in mapped if t["type"] == "T-ent")
|
||||
print(f"=== guard-5 judge projection (WIN_CHARS={WIN_CHARS}, tent_control={tent_control}) ===")
|
||||
print(f"subset {len(sub)} flat-mappable {n} (T-ana {sum(1 for t in mapped if t['type']=='T-ana')}, "
|
||||
f"T-ell {sum(1 for t in mapped if t['type']=='T-ell')}, T-ent {sum(1 for t in mapped if t['type']=='T-ent')}) "
|
||||
f"oracle-mappable {q1b_ok}")
|
||||
print(f"per-cell in_tok~{in_tok} votes~{votes} cost/cell ~${cc:.4f}")
|
||||
print(f"Q1b(re-judge) cells {q1b_ok} -> ~${q1b_ok*cc:.2f}")
|
||||
print(f"FLOOR cells {n} -> ~${n*cc:.2f}")
|
||||
print(f"Q3a cells {q3a_cells} -> ~${q3a_cells*cc:.2f} | Q0 cells {n} -> ~${n*cc:.2f}")
|
||||
|
||||
|
||||
def run(block, tent_control, per_call_cap, limit=0, full=False):
|
||||
flat_src, oracle_src, flat_fin, oracle_fin, traps = load()
|
||||
primary_only = block in ("FLOOR",)
|
||||
sub = select_traps(traps, tent_control, primary_only=primary_only)
|
||||
if limit:
|
||||
sub = sub[:limit]
|
||||
a0p_fin = db_finals(AH / "flat_a0prime_30.db") if block == "FLOOR" else None
|
||||
draft_fin = db_drafts(AH / "flat_a0_30.db") if block == "Q0" else None
|
||||
# GLOBAL cap: seed this block's Spender with spend already booked in all OTHER ledgers so hard_cap is
|
||||
# a true cross-block ceiling (review I8), not a per-block one that would sum to ~4x the freeze.
|
||||
seed = global_spent_excluding(block)
|
||||
sp = L.Spender(str(SD / f"judge_costs_{block}.jsonl"), per_call_cap=per_call_cap,
|
||||
hard_cap=GLOBAL_CEILING, seed_total=seed)
|
||||
print(f"[budget] global spent (other ledgers) ${seed:.3f}; block start ${sp.total:.3f}; "
|
||||
f"global ceiling ${GLOBAL_CEILING}; excerpt={'FULL-CHUNK' if full else f'window{WIN_CHARS}'}")
|
||||
out, skipped = [], []
|
||||
for t in sub:
|
||||
m = map_trap(t, flat_src, oracle_src, flat_fin, oracle_fin, full=full)
|
||||
if not m:
|
||||
skipped.append((t["id"], "flat-unmappable")); continue
|
||||
dep = m["flat_chunk"]
|
||||
if block == "Q1b":
|
||||
if not m["oracle"]:
|
||||
skipped.append((t["id"], "oracle-unmappable")); continue
|
||||
agg = judge_cell(sp, m["jt"], a0_text=m["flat_win"], arm_text=m["oracle"]["win"])
|
||||
agg.update(block="Q1b", arm="oracle", flat_chunk=dep, flat_frac=m["flat_frac"],
|
||||
oracle_chunk=m["oracle"]["chunk"], oracle_frac=m["oracle"]["frac"], full=full)
|
||||
out.append(agg)
|
||||
print(f" Q1b {t['id']}: oracle-flat {agg['paired_diff_arm_minus_a0']} "
|
||||
f"(ffrac {m['flat_frac']} ofrac {m['oracle']['frac']} trunc={agg.get('budget_truncated')}) spend ${sp.total:.3f}")
|
||||
elif block == "FLOOR":
|
||||
if dep not in a0p_fin:
|
||||
skipped.append((t["id"], "a0prime-missing")); continue
|
||||
agg = judge_cell(sp, m["jt"], a0_text=m["flat_win"],
|
||||
arm_text=excerpt(a0p_fin[dep], m["flat_frac"], full))
|
||||
agg.update(block="FLOOR", arm="a0prime", flat_chunk=dep, flat_frac=m["flat_frac"], full=full)
|
||||
out.append(agg)
|
||||
print(f" FLOOR {t['id']}: |diff| {abs(agg['paired_diff_arm_minus_a0'] or 0):.3f} spend ${sp.total:.3f}")
|
||||
elif block == "Q3a":
|
||||
modes = ["q3a-src", "q3a-draft", "q3a-final"] if t["type"] in ("T-ana", "T-ell") else ["q3a-final"]
|
||||
for mode in modes:
|
||||
af = arm_final(mode, dep)
|
||||
if not af:
|
||||
skipped.append((t["id"], f"{mode}-missing")); continue
|
||||
agg = judge_cell(sp, m["jt"], a0_text=m["flat_win"], arm_text=excerpt(af, m["flat_frac"], full))
|
||||
agg.update(block="Q3a", arm=mode, flat_chunk=dep, flat_frac=m["flat_frac"], full=full)
|
||||
out.append(agg)
|
||||
print(f" Q3a {t['id']} {mode}: {agg['paired_diff_arm_minus_a0']} spend ${sp.total:.3f}")
|
||||
elif block == "Q0":
|
||||
if dep not in draft_fin:
|
||||
skipped.append((t["id"], "draft-missing")); continue
|
||||
agg = judge_cell(sp, m["jt"], a0_text=excerpt(draft_fin[dep], m["flat_frac"], full),
|
||||
arm_text=m["flat_win"]) # a0=do-nothing(draft), arm=A0(editor final)
|
||||
agg.update(block="Q0", arm="A0-editor", flat_chunk=dep, flat_frac=m["flat_frac"], full=full)
|
||||
out.append(agg)
|
||||
print(f" Q0 {t['id']}: A0-donothing {agg['paired_diff_arm_minus_a0']} spend ${sp.total:.3f}")
|
||||
if any(a.get("budget_truncated") for a in out):
|
||||
print(f" [HARD-CAP] budget ceiling ${GLOBAL_CEILING} bound mid-run -> stopping block"); break
|
||||
Path(SD / f"judge_results_{block}.json").write_text(
|
||||
json.dumps({"n": len(out), "win_chars": (None if full else WIN_CHARS), "full_chunk": full,
|
||||
"skipped": skipped, "results": out}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\n{block}: {len(out)} cells judged, {len(skipped)} skipped {skipped}, block+global spend ${sp.total:.4f}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--project", action="store_true")
|
||||
ap.add_argument("--run", choices=["Q1b", "Q3a", "Q0", "FLOOR"])
|
||||
ap.add_argument("--tent-control", type=int, default=10)
|
||||
ap.add_argument("--per-call-cap", type=float, default=0.03)
|
||||
ap.add_argument("--full", action="store_true", help="feed WHOLE finals (no windowing) — used for Q1b")
|
||||
ap.add_argument("--limit", type=int, default=0, help="judge only first N traps (selftest)")
|
||||
a = ap.parse_args()
|
||||
traps = json.load(open(SD / "final_traps.json"))["traps"]
|
||||
if a.project:
|
||||
project(traps, a.tent_control)
|
||||
elif a.run:
|
||||
run(a.run, a.tent_control, a.per_call_cap, a.limit, full=a.full)
|
||||
else:
|
||||
ap.print_help()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
91
eval/exp15/judge_selftest.py
Normal file
91
eval/exp15/judge_selftest.py
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — judge-harness SELFTEST on 2-3 pilot traps (checkpoint item d + preliminary FLOOR).
|
||||
|
||||
Judges A0 vs A0' (SAME chunking = the noise floor: a well-behaved harness should return ~tie, low
|
||||
|paired_diff|, no systematic order bias). Exercises: JSON parse-rate, both-orders, the 6->10 split
|
||||
escalation branch, leave-one-judge-out, catastrophe screen. Cheap (cents).
|
||||
|
||||
Pilot traps are hand-specified cohesion phenomena located in specific chunks; each is judged on that
|
||||
chunk's A0-final vs A0'-final. Run AFTER anchors_analyze.py (needs *.chunks.json).
|
||||
Run: python judge_selftest.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import exp15_llm as L
|
||||
from judges import judge_cell
|
||||
|
||||
AH = Path("/home/ubuntu/books/gu-zhenren/exp15/anchors")
|
||||
TRAP_LEDGER = Path("/home/ubuntu/books/gu-zhenren/exp15/trap_ledger.json")
|
||||
LEDGER = Path("/home/ubuntu/books/gu-zhenren/exp15/judge_selftest_costs.jsonl")
|
||||
OUT = Path("/home/ubuntu/books/gu-zhenren/exp15/judge_selftest.json")
|
||||
A0_TAG = "flat_a0" # blind-greedy anchor (regenerated on flat S2')
|
||||
A0P_TAG = "flat_a0prime"
|
||||
|
||||
|
||||
def chunks(tag):
|
||||
return {c["chunk"]: c for c in json.load(open(AH / f"{tag}.chunks.json"))}
|
||||
|
||||
|
||||
def locate_chunk(chunk_map, src_probe):
|
||||
"""Return the chunk_idx whose source contains src_probe (first ~20 chars)."""
|
||||
probe = src_probe.strip()[:20]
|
||||
for idx, c in chunk_map.items():
|
||||
if c["source"] and probe in c["source"]:
|
||||
return idx
|
||||
return None
|
||||
|
||||
|
||||
def pick_pilots(n=3):
|
||||
"""2-3 diverse-type traps from the curated ledger (prefer distinct types crossing an A0 cut)."""
|
||||
led = json.load(open(TRAP_LEDGER))["ledger"]
|
||||
cross = [t for t in led if t.get("crosses_A0_cut")]
|
||||
picks, seen_types = [], set()
|
||||
for t in cross:
|
||||
if t["type"] not in seen_types:
|
||||
picks.append(t); seen_types.add(t["type"])
|
||||
if len(picks) >= n:
|
||||
break
|
||||
return picks or led[:n]
|
||||
|
||||
|
||||
def main():
|
||||
a0 = chunks(A0_TAG)
|
||||
a0p = chunks(A0P_TAG)
|
||||
pilots = [{"id": t["id"], "type": t["type"], "chunk_idx": locate_chunk(a0, t["anchor_span"]),
|
||||
"src_zone": (t["anchor_span"] + " … " + t["dependent_span"]),
|
||||
"phenomenon": t["phenomenon"], "expected": t["expected_ru"]}
|
||||
for t in pick_pilots(3)]
|
||||
pilots = [p for p in pilots if p["chunk_idx"] is not None]
|
||||
sp = L.Spender(str(LEDGER), per_call_cap=0.05, hard_cap=1.0)
|
||||
results = []
|
||||
for p in pilots:
|
||||
ck = p["chunk_idx"]
|
||||
a0_final = a0[ck]["final"]
|
||||
armp_final = a0p[ck]["final"]
|
||||
trap = {"id": p["id"], "type": p["type"], "src_zone": p["src_zone"],
|
||||
"phenomenon": p["phenomenon"], "expected": p["expected"]}
|
||||
print(f"\n=== pilot {p['id']} ({p['type']}) chunk {ck} — A0 vs A0' (floor) ===")
|
||||
agg = judge_cell(sp, trap, a0_final, armp_final)
|
||||
results.append(agg)
|
||||
print(json.dumps({k: agg[k] for k in ("n_votes", "parse_fail", "a0_accuracy", "arm_accuracy",
|
||||
"paired_diff_arm_minus_a0", "pairwise", "order_bias", "catastrophe")}, ensure_ascii=False, indent=2))
|
||||
OUT.write_text(json.dumps({"pilots": results, "total_cost": round(sp.total, 5)}, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8")
|
||||
# selftest checks
|
||||
tot_parse_fail = sum(r["parse_fail"] for r in results)
|
||||
tot_votes = sum(r["n_votes"] for r in results)
|
||||
escalated = any(r["n_votes"] > 2 * 6 for r in results) # >12 -> at least one judge escalated
|
||||
print(f"\n=== SELFTEST SUMMARY ===")
|
||||
print(f"total votes {tot_votes} parse_fail {tot_parse_fail} (want 0) spend ${sp.total:.4f}")
|
||||
print(f"escalation branch exercised (some cell >12 votes): {escalated}")
|
||||
print(f"floor |paired_diff| per pilot: {[abs(r['paired_diff_arm_minus_a0'] or 0) for r in results]} "
|
||||
f"(want small — A0 vs A0' is noise)")
|
||||
print(f"-> {OUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
242
eval/exp15/judges.py
Normal file
242
eval/exp15/judges.py
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — pairwise span-citing trap judge (design §D3 / §1.6). Built from scratch: the exp14 harness
|
||||
has NEITHER both-orders NOR 6->10 escalation.
|
||||
|
||||
Per trap-cell (one trap × one A0-vs-arm contrast), each of >=2 cross-family judges (grok-4.3 reasoning-ON
|
||||
+ mistral-large-latest) is asked, BLIND (V1/V2), in BOTH orders, REP times, whether each translation
|
||||
correctly resolves the SPECIFIC cohesion phenomenon (cite a <=15-word span). We record per-arm verdicts
|
||||
(correct/partial/wrong -> accuracy) AND the pairwise winner AND catastrophe flags. Aggregation:
|
||||
base 6 votes/judge (3 reps × 2 orders); if a judge's decisive split is <=4:2 -> +4 to 10 (pre-reg rule,
|
||||
budget-relaxed from §A.7 "~10", fixed explicitly). Leave-one-judge-out. Catastrophe screen to the winner too.
|
||||
Primary Q1/Q3 metric = per-arm ACCURACY (correct=1/partial=0.5/wrong=0), paired diff arm-A0. Pairwise
|
||||
"better" is a robustness cross-check.
|
||||
|
||||
temp 0 (grok reasoning-ON is mildly stochastic across reps; mistral deterministic -> reps agree = strong
|
||||
consensus, not wasted signal). Every call persists usage+cost via exp15_llm.Spender; per-call gate on.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import exp15_llm as L
|
||||
|
||||
JUDGES = ["grok-4.3", "mistral-large-latest"]
|
||||
BASE_REPS_PER_ORDER = 3 # 3 reps × 2 orders = 6 base votes/judge
|
||||
ESCALATE_REPS_PER_ORDER = 2 # +2 reps × 2 orders = +4 -> 10 on split
|
||||
SCORE = {"correct": 1.0, "partial": 0.5, "wrong": 0.0}
|
||||
|
||||
JUDGE_SYS = (
|
||||
"Ты — придирчивый билингвальный (китайский↔русский) литературный рецензент. Тебе дают ОДИН конкретный "
|
||||
"переводческий феномен связности (анафора, время/род глагола, катафора, эллипсис, перенос сущности через "
|
||||
"границу и т.п.), фрагмент КИТАЙСКОГО исходника и ДВА перевода этого места — V1 и V2. Оцени СТРОГО и ТОЛЬКО "
|
||||
"заданный феномен (не общий стиль). Для каждого перевода реши: correct (феномен передан верно), partial "
|
||||
"(частично/неоднозначно), wrong (искажён/потерян/неверный род-время-референт). Затем реши, какой перевод "
|
||||
"лучше справился ИМЕННО с этим феноменом: V1, V2 или tie. Отметь catastrophe=true, если перевод даёт грубую "
|
||||
"смысловую ошибку по этому феномену (неверный род/референт/полярность/потеря). Приведи доказательство — "
|
||||
"цитату <=15 слов из каждого перевода. Выведи СТРОГО JSON без пояснений:\n"
|
||||
'{"v1":"correct|partial|wrong","v2":"correct|partial|wrong","better":"V1|V2|tie",'
|
||||
'"cat_v1":true|false,"cat_v2":true|false,"ev_v1":"...","ev_v2":"..."}'
|
||||
)
|
||||
|
||||
|
||||
def _user_prompt(trap, v1_text, v2_text):
|
||||
return (
|
||||
f"ФЕНОМЕН СВЯЗНОСТИ ({trap['type']}): {trap['phenomenon']}\n"
|
||||
f"ЧТО ТРЕБУЕТСЯ ДЛЯ ВЕРНОЙ ПЕРЕДАЧИ: {trap['expected']}\n\n"
|
||||
f"КИТАЙСКИЙ ИСХОДНИК (место феномена):\n{trap['src_zone']}\n\n"
|
||||
f"V1 (перевод этого места):\n{v1_text}\n\n"
|
||||
f"V2 (перевод этого места):\n{v2_text}\n\n"
|
||||
"Оцени ТОЛЬКО указанный феномен. Ответ — строго JSON по схеме."
|
||||
)
|
||||
|
||||
|
||||
_JSON_RE = re.compile(r"\{.*\}", re.S)
|
||||
|
||||
|
||||
def _as_bool(x):
|
||||
"""Robust truthiness for the catastrophe flags. A judge that emits the boolean as a JSON STRING
|
||||
("cat_v1":"false") would otherwise become bool("false")==True -> a spurious catastrophe (review I4)."""
|
||||
return x is True or (isinstance(x, str) and x.strip().lower() in ("true", "1", "yes"))
|
||||
|
||||
|
||||
def parse_verdict(text):
|
||||
m = _JSON_RE.search(text or "")
|
||||
if not m:
|
||||
return None
|
||||
try:
|
||||
d = json.loads(m.group(0))
|
||||
except Exception:
|
||||
# tolerant: pull ALL fields by regex (incl cat flags + evidence — review I4: the old fallback
|
||||
# dropped cat/ev, silently zeroing a flagged catastrophe on a parse-degraded but non-empty vote)
|
||||
d = {}
|
||||
for k in ("v1", "v2", "better"):
|
||||
mm = re.search(rf'"{k}"\s*:\s*"(\w+)"', text)
|
||||
if mm:
|
||||
d[k] = mm.group(1)
|
||||
for k in ("cat_v1", "cat_v2"):
|
||||
mm = re.search(rf'"{k}"\s*:\s*(true|false|"true"|"false")', text, re.I)
|
||||
if mm:
|
||||
d[k] = mm.group(1)
|
||||
for k in ("ev_v1", "ev_v2"):
|
||||
mm = re.search(rf'"{k}"\s*:\s*"([^"]*)"', text)
|
||||
if mm:
|
||||
d[k] = mm.group(1)
|
||||
for k in ("v1", "v2"):
|
||||
if d.get(k) not in SCORE:
|
||||
return None
|
||||
if d.get("better") not in ("V1", "V2", "tie"):
|
||||
d["better"] = "tie"
|
||||
d["cat_v1"] = _as_bool(d.get("cat_v1", False))
|
||||
d["cat_v2"] = _as_bool(d.get("cat_v2", False))
|
||||
return d
|
||||
|
||||
|
||||
@dataclass
|
||||
class Vote:
|
||||
judge: str
|
||||
order: str # "AB" (a0=V1, arm=V2) or "BA" (arm=V1, a0=V2)
|
||||
a0_verdict: str # correct/partial/wrong
|
||||
arm_verdict: str
|
||||
winner: str # "a0" | "arm" | "tie"
|
||||
a0_cat: bool
|
||||
arm_cat: bool
|
||||
ev_a0: str = "" # judge's <=15-word evidence span for the a0 side (order-normalised)
|
||||
ev_arm: str = "" # judge's evidence span for the arm side
|
||||
raw: dict = field(default_factory=dict)
|
||||
|
||||
|
||||
# a judge verdict is a tiny JSON (~60-150 completion tokens); grok reasoning is separate/uncapped by
|
||||
# max_tokens. Cap max_tokens LOW so predict_cost's (max_out + reasoning_ceil) gate estimate stays under
|
||||
# the judge per-call cap — otherwise grok's 8000-default made predict ~$0.033 > $0.03 and grok was
|
||||
# silently gate-blocked (0 grok votes; the bug this selftest caught).
|
||||
JUDGE_MAX_TOKENS = 1500
|
||||
|
||||
|
||||
def _one_call(sp: L.Spender, judge, trap, v1_text, v2_text):
|
||||
"""Return (verdict|None, rec, status). status in {ok, gate_block, err, parse_fail} so the caller can
|
||||
tell an out-of-budget refusal (gate_block) from a model parse failure (review I8)."""
|
||||
msgs = [{"role": "system", "content": JUDGE_SYS},
|
||||
{"role": "user", "content": _user_prompt(trap, v1_text, v2_text)}]
|
||||
in_est = int((len(JUDGE_SYS) + len(msgs[1]["content"])) / 1.9) # conservative chars/token (review NIT)
|
||||
ok, pred, why = sp.gate(judge, in_est, JUDGE_MAX_TOKENS)
|
||||
if not ok:
|
||||
return None, {"judge": judge, "trap": trap["id"], "err": why, "cost": 0.0}, "gate_block"
|
||||
r = L.call(judge, msgs, temp=0.0, max_tokens=JUDGE_MAX_TOKENS)
|
||||
rec = {"judge": judge, "trap": trap["id"], "usage": r.get("usage"), "cost": r.get("cost", 0.0),
|
||||
"err": r.get("err"), "finish": r.get("finish"), "latency_s": r.get("latency_s"),
|
||||
"retry_fails": r.get("retry_fails")}
|
||||
sp.record(rec)
|
||||
import os
|
||||
if r.get("err"):
|
||||
if os.environ.get("JUDGE_DEBUG"):
|
||||
print(f" [DBG {judge}] ERR {r['err'][:70]}")
|
||||
return None, rec, "err"
|
||||
v = parse_verdict(r["text"])
|
||||
if v is None:
|
||||
if os.environ.get("JUDGE_DEBUG"):
|
||||
print(f" [DBG {judge}] NOPARSE content[{len(r.get('text') or '')}]={repr((r.get('text') or '')[:80])} "
|
||||
f"finish={r.get('finish')} reasoning[{len(r.get('reasoning') or '')}]")
|
||||
return None, rec, "parse_fail"
|
||||
return v, rec, "ok"
|
||||
|
||||
|
||||
def judge_cell(sp: L.Spender, trap, a0_text, arm_text, judges=JUDGES,
|
||||
base_reps=BASE_REPS_PER_ORDER, esc_reps=ESCALATE_REPS_PER_ORDER):
|
||||
"""Return per-cell aggregate with per-arm accuracy, pairwise, catastrophe, per-judge (for LOO)."""
|
||||
votes: list[Vote] = []
|
||||
parse_fail = 0
|
||||
budget_truncated = False
|
||||
for judge in judges:
|
||||
if budget_truncated:
|
||||
break
|
||||
jvotes = []
|
||||
def do_reps(reps):
|
||||
nonlocal parse_fail, budget_truncated
|
||||
for _ in range(reps):
|
||||
if budget_truncated:
|
||||
return
|
||||
for order in ("AB", "BA"):
|
||||
if order == "AB":
|
||||
v1, v2 = a0_text, arm_text
|
||||
else:
|
||||
v1, v2 = arm_text, a0_text
|
||||
d, rec, status = _one_call(sp, judge, trap, v1, v2)
|
||||
if status == "gate_block":
|
||||
budget_truncated = True # out of budget -> stop this cell, do NOT score partial
|
||||
return
|
||||
if d is None:
|
||||
parse_fail += 1
|
||||
continue
|
||||
if order == "AB":
|
||||
a0v, armv = d["v1"], d["v2"]
|
||||
win = {"V1": "a0", "V2": "arm", "tie": "tie"}[d["better"]]
|
||||
a0c, armc = bool(d["cat_v1"]), bool(d["cat_v2"])
|
||||
eva0, evarm = d.get("ev_v1", ""), d.get("ev_v2", "")
|
||||
else:
|
||||
a0v, armv = d["v2"], d["v1"]
|
||||
win = {"V1": "arm", "V2": "a0", "tie": "tie"}[d["better"]]
|
||||
a0c, armc = bool(d["cat_v2"]), bool(d["cat_v1"])
|
||||
eva0, evarm = d.get("ev_v2", ""), d.get("ev_v1", "")
|
||||
jvotes.append(Vote(judge, order, a0v, armv, win, a0c, armc, eva0, evarm, d))
|
||||
do_reps(base_reps)
|
||||
# escalation: if this judge's decisive split is <=4:2 (winner has <=4 of 6 decisive), add reps
|
||||
dec = [v for v in jvotes if v.winner in ("a0", "arm")]
|
||||
arm_w = sum(1 for v in dec if v.winner == "arm")
|
||||
a0_w = sum(1 for v in dec if v.winner == "a0")
|
||||
if not budget_truncated and max(arm_w, a0_w) <= 4: # split <=4:2 (or ties dominate) -> escalate
|
||||
do_reps(esc_reps)
|
||||
votes.extend(jvotes)
|
||||
return _aggregate(trap, votes, parse_fail, judges, budget_truncated)
|
||||
|
||||
|
||||
def _acc(verdicts):
|
||||
return sum(SCORE[v] for v in verdicts) / len(verdicts) if verdicts else None
|
||||
|
||||
|
||||
def _aggregate(trap, votes, parse_fail, judges, budget_truncated=False):
|
||||
a0_acc = _acc([v.a0_verdict for v in votes])
|
||||
arm_acc = _acc([v.arm_verdict for v in votes])
|
||||
arm_w = sum(1 for v in votes if v.winner == "arm")
|
||||
a0_w = sum(1 for v in votes if v.winner == "a0")
|
||||
tie = sum(1 for v in votes if v.winner == "tie")
|
||||
# leave-one-judge-out on the paired accuracy diff
|
||||
loo = {}
|
||||
for jx in judges:
|
||||
sub = [v for v in votes if v.judge != jx]
|
||||
if sub:
|
||||
loo[f"without_{jx}"] = round((_acc([v.arm_verdict for v in sub]) or 0)
|
||||
- (_acc([v.a0_verdict for v in sub]) or 0), 3)
|
||||
# order-bias check: winner distribution per order
|
||||
order_bias = {}
|
||||
for o in ("AB", "BA"):
|
||||
ov = [v for v in votes if v.order == o]
|
||||
order_bias[o] = {"arm": sum(1 for v in ov if v.winner == "arm"),
|
||||
"a0": sum(1 for v in ov if v.winner == "a0"),
|
||||
"tie": sum(1 for v in ov if v.winner == "tie")}
|
||||
return {
|
||||
"trap": trap["id"], "type": trap["type"],
|
||||
"n_votes": len(votes), "parse_fail": parse_fail, "budget_truncated": budget_truncated,
|
||||
"a0_accuracy": round(a0_acc, 3) if a0_acc is not None else None,
|
||||
"arm_accuracy": round(arm_acc, 3) if arm_acc is not None else None,
|
||||
"paired_diff_arm_minus_a0": round((arm_acc or 0) - (a0_acc or 0), 3) if a0_acc is not None else None,
|
||||
"pairwise": {"arm_wins": arm_w, "a0_wins": a0_w, "tie": tie},
|
||||
"catastrophe": {"a0": sum(1 for v in votes if v.a0_cat), "arm": sum(1 for v in votes if v.arm_cat)},
|
||||
"leave_one_judge_out": loo,
|
||||
"order_bias": order_bias,
|
||||
"per_judge": {j: {"arm_acc": round(_acc([v.arm_verdict for v in votes if v.judge == j]) or 0, 3),
|
||||
"a0_acc": round(_acc([v.a0_verdict for v in votes if v.judge == j]) or 0, 3),
|
||||
"n": sum(1 for v in votes if v.judge == j)} for j in judges},
|
||||
# per-vote raw verdicts (a0/arm-normalised, order recorded) — persisted so catastrophe/LOO/order
|
||||
# audits run on the actual votes, not just the aggregate counts (orchestrator D30.10 fix).
|
||||
"votes": [{"judge": v.judge, "order": v.order, "a0": v.a0_verdict, "arm": v.arm_verdict,
|
||||
"winner": v.winner, "a0_cat": v.a0_cat, "arm_cat": v.arm_cat,
|
||||
"ev_a0": v.ev_a0, "ev_arm": v.ev_arm} for v in votes],
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("judge harness module — import judge_cell(). Run selftest via judge_selftest.py.")
|
||||
508
eval/exp15/mem_select.py
Normal file
508
eval/exp15/mem_select.py
Normal file
|
|
@ -0,0 +1,508 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — byte-faithful Python PORT of the backend memory-bank hot path (memory.go + memnorm.go).
|
||||
|
||||
Reconstructs the injection the Go pipeline actually feeds per chunk: normalize -> Aho-Corasick
|
||||
(here: exact multi-substring, same result set) -> spaced-phonetic boundary drop -> trust-gated
|
||||
longest-match suppression -> spoiler window -> three-way disposition -> sticky scene-inertia (depth 2)
|
||||
-> priority token budget -> render (translator glossary block / editor constraint block).
|
||||
|
||||
WHY a port (orchestrator verdict 2026-07-17): retrieval_state.injected_ids carries EXACT hits only
|
||||
(sticky-blind by construction), so faithful arm injection needs the full Select, VALIDATED against Go's
|
||||
actually-rendered glossary block (LOG_LLM_BODIES byte-diff), NOT against injected_ids. On divergence:
|
||||
trust Go (fix the port). This module is the injection engine shared by A0-validation and all arms.
|
||||
|
||||
Source of truth mirrored (backend/internal/pipeline/):
|
||||
memnorm.go normalizeSourceKey / significantLen / trad2simp
|
||||
memory.go Select / suppressContained / matchTrust / trustRank / dispositionFor / spoilerBlocked
|
||||
/ priorityRank / glossaryLineTokens / renderGlossaryBlock / renderEditorConstraintBlock
|
||||
/ stickyDepth=2 / minKeyLenHan=2 / minKeyLenPhonetic=3
|
||||
memseed.go loadGlossarySeed (status ""->approved; surfaces = src + aliases when dst non-empty)
|
||||
|
||||
Self-review by EXECUTION: run `python mem_select.py --selftest` for invariants; the real gate is the
|
||||
byte-diff vs Go's logged block in validate_injection.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import unicodedata
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import regex # \p{Han} etc. — exact Unicode script tables, matching Go's unicode.Han/Hiragana/...
|
||||
import yaml
|
||||
|
||||
BACKEND = Path("/home/ubuntu/projects/textmachine/backend")
|
||||
TRAD2SIMP_FILE = BACKEND / "internal/pipeline/data/trad2simp.txt"
|
||||
|
||||
MIN_KEY_LEN_HAN = 2
|
||||
MIN_KEY_LEN_PHONETIC = 3
|
||||
STICKY_DEPTH = 2
|
||||
|
||||
# disposition constants (memory.go:89-93)
|
||||
CONFIRMED = "confirmed"
|
||||
AMBIGUOUS = "ambiguous"
|
||||
REJECT = "reject"
|
||||
|
||||
GLOSSARY_HEADER = ("ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; "
|
||||
"строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):")
|
||||
EDITOR_HEADER = ("КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике эти сущности должны быть переданы "
|
||||
"ИМЕННО этими формами — приводи к ним любые расхождения, склоняя по контексту; не вводи "
|
||||
"иных вариантов и не меняй ничего другого):")
|
||||
|
||||
# --- Unicode script predicates (match Go unicode.Han|Hiragana|Katakana|Hangul + IsLetter/Cf) --------
|
||||
_RE_CJK = regex.compile(r"[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]")
|
||||
_RE_LETTER = regex.compile(r"\p{L}")
|
||||
_RE_HAN = regex.compile(r"\p{Han}")
|
||||
_RE_LATIN = regex.compile(r"\p{Latin}")
|
||||
_RE_CYRILLIC = regex.compile(r"\p{Cyrillic}")
|
||||
|
||||
|
||||
def _is_cjk(ch: str) -> bool:
|
||||
return _RE_CJK.match(ch) is not None
|
||||
|
||||
|
||||
def _is_han(ch: str) -> bool:
|
||||
return _RE_HAN.match(ch) is not None
|
||||
|
||||
|
||||
def _is_letter(ch: str) -> bool:
|
||||
return _RE_LETTER.match(ch) is not None
|
||||
|
||||
|
||||
def _is_apostrophe_variant(ch: str) -> bool:
|
||||
return ch in ("‘", "’", "ʼ", "＇") # ' ' ʼ ＇ (memnorm.go:189-195)
|
||||
|
||||
|
||||
def _is_ignorable_for_match(ch: str) -> bool:
|
||||
# unicode.Cf OR variation selectors (memnorm.go:207-212)
|
||||
if unicodedata.category(ch) == "Cf":
|
||||
return True
|
||||
o = ord(ch)
|
||||
return (0xFE00 <= o <= 0xFE0F) or (0xE0100 <= o <= 0xE01EF)
|
||||
|
||||
|
||||
# --- trad2simp table (loaded from the Go embed file, parsed identically) ----------------------------
|
||||
def _load_trad2simp() -> dict:
|
||||
m = {}
|
||||
for line in TRAD2SIMP_FILE.read_text(encoding="utf-8").split("\n"):
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
fields = line.split()
|
||||
if len(fields) != 2:
|
||||
raise ValueError(f"trad2simp corrupt line: {line!r}")
|
||||
tr, si = fields
|
||||
if len(tr) != 1 or len(si) != 1:
|
||||
raise ValueError(f"trad2simp non-single-rune: {line!r}")
|
||||
m[tr] = si
|
||||
return m
|
||||
|
||||
|
||||
_TRAD2SIMP = _load_trad2simp()
|
||||
|
||||
|
||||
def normalize_source_key(s: str) -> str:
|
||||
"""memnorm.go:104 — NFKC -> apostrophe fold -> drop ignorables -> trad2simp -> kana->hira -> lower."""
|
||||
s = unicodedata.normalize("NFKC", s)
|
||||
out = []
|
||||
for ch in s:
|
||||
if _is_apostrophe_variant(ch):
|
||||
ch = "'"
|
||||
if _is_ignorable_for_match(ch):
|
||||
continue
|
||||
ch = _TRAD2SIMP.get(ch, ch)
|
||||
o = ord(ch)
|
||||
if 0x30A1 <= o <= 0x30F6: # katakana -> hiragana
|
||||
ch = chr(o - 0x60)
|
||||
out.append(ch.lower())
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def significant_len(normalized: str) -> int:
|
||||
"""memnorm.go:220 — count CJK/kana/Hangul ideographs + letters."""
|
||||
n = 0
|
||||
for ch in normalized:
|
||||
if _is_cjk(ch) or _is_letter(ch):
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def runes_any_han(s: str) -> bool:
|
||||
return _RE_HAN.search(s) is not None
|
||||
|
||||
|
||||
def min_key_len_for(norm_key: str) -> int:
|
||||
return MIN_KEY_LEN_HAN if runes_any_han(norm_key) else MIN_KEY_LEN_PHONETIC
|
||||
|
||||
|
||||
def collision_prone_key(norm_key: str) -> bool:
|
||||
return (not runes_any_han(norm_key)) and significant_len(norm_key) <= MIN_KEY_LEN_PHONETIC
|
||||
|
||||
|
||||
def glossary_line_tokens(src: str, dst: str) -> int:
|
||||
"""memory.go:304 — cjk + other//3 over 'src → dst' (floor-free)."""
|
||||
cjk = other = 0
|
||||
for ch in f"{src} → {dst}":
|
||||
if _is_cjk(ch):
|
||||
cjk += 1
|
||||
elif ch.isspace():
|
||||
pass
|
||||
else:
|
||||
other += 1
|
||||
return cjk + other // 3
|
||||
|
||||
|
||||
def trust_rank(d: str) -> int:
|
||||
return {CONFIRMED: 2, AMBIGUOUS: 1}.get(d, 0)
|
||||
|
||||
|
||||
# --- data model -------------------------------------------------------------------------------------
|
||||
@dataclass
|
||||
class MemoryEntry:
|
||||
id: str
|
||||
src: str
|
||||
dst: str
|
||||
status: str
|
||||
sense: str
|
||||
since_ch: int
|
||||
until_ch: int
|
||||
norm_keys: list = field(default_factory=list)
|
||||
allow_short: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class Picked:
|
||||
entry: MemoryEntry
|
||||
via: str # firing key, or "sticky"
|
||||
disp: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class Selection:
|
||||
injected: list = field(default_factory=list) # [Picked] in budget priority order
|
||||
rejected: list = field(default_factory=list)
|
||||
evicted: list = field(default_factory=list)
|
||||
trust_gated: list = field(default_factory=list)
|
||||
active_ids: dict = field(default_factory=dict) # id -> disp (EXACT only; feeds next sticky)
|
||||
|
||||
|
||||
class MemoryBank:
|
||||
"""Materialized frozen glossary. Built once from seed rows; never mutated (memory.go:180)."""
|
||||
|
||||
def __init__(self, rows: list):
|
||||
self.entries: list[MemoryEntry] = []
|
||||
self.key_owners: dict[str, list[int]] = {}
|
||||
seen_key = set()
|
||||
all_keys = []
|
||||
# frozen entry order = store's GlossaryForBook: ORDER BY src, sense, since_ch, until_ch, status, dst
|
||||
# (SQLite BINARY collation == UTF-8 byte order == codepoint order for BMP CJK). This fixes the
|
||||
# injection BLOCK line order (the id SET already matched; only the order differed).
|
||||
rows = sorted(rows, key=lambda r: (r["src"], r.get("sense") or "",
|
||||
int(r.get("since_ch", 0) or 0), int(r.get("until_ch", 0) or 0),
|
||||
r.get("status") or "approved", r.get("dst") or ""))
|
||||
for row in rows:
|
||||
src = row["src"]
|
||||
sense = row.get("sense", "") or ""
|
||||
since = int(row.get("since_ch", 0) or 0)
|
||||
until = int(row.get("until_ch", 0) or 0)
|
||||
status = row.get("status") or "approved" # memseed.go:100-102
|
||||
allow_short = bool(row.get("allow_short", False))
|
||||
e = MemoryEntry(
|
||||
id=f"{src}\x1f{sense}\x1f{since}\x1f{until}",
|
||||
src=src, dst=row.get("dst", "") or "", status=status,
|
||||
sense=sense, since_ch=since, until_ch=until, allow_short=allow_short,
|
||||
)
|
||||
# surfaces = src + aliases, ONLY when dst non-empty (memory.go:202-208)
|
||||
surfaces = []
|
||||
if (e.dst or "").strip() != "":
|
||||
surfaces.append(src)
|
||||
for a in (row.get("aliases") or []):
|
||||
al = (a.get("alias") or "").strip()
|
||||
if al:
|
||||
surfaces.append(al)
|
||||
for s in surfaces:
|
||||
nk = normalize_source_key(s)
|
||||
if nk == "":
|
||||
continue
|
||||
if significant_len(nk) < min_key_len_for(nk) and not allow_short:
|
||||
continue # single-key ban
|
||||
e.norm_keys.append(nk)
|
||||
idx = len(self.entries)
|
||||
self.entries.append(e)
|
||||
for nk in e.norm_keys:
|
||||
self.key_owners.setdefault(nk, []).append(idx)
|
||||
if nk not in seen_key:
|
||||
seen_key.add(nk)
|
||||
all_keys.append(nk)
|
||||
self.keys = sorted(all_keys) # deterministic (memory.go:241)
|
||||
self.key_index = {k: i for i, k in enumerate(self.keys)}
|
||||
|
||||
# ---- matching (AC-equivalent exact multi-substring over normalized text) ----
|
||||
def _matches(self, ntext: str):
|
||||
"""Return [(key_idx, start, end)] for every occurrence of every key, sorted (start,end,key_idx).
|
||||
Overlapping occurrences included (pos advances by 1), matching AC output exactly."""
|
||||
out = []
|
||||
for k in self.keys:
|
||||
ki = self.key_index[k]
|
||||
klen = len(k)
|
||||
pos = 0
|
||||
while True:
|
||||
i = ntext.find(k, pos)
|
||||
if i < 0:
|
||||
break
|
||||
out.append((ki, i, i + klen))
|
||||
pos = i + 1
|
||||
out.sort(key=lambda m: (m[1], m[2], m[0]))
|
||||
return out
|
||||
|
||||
def _spaced_phonetic_script(self, norm_key: str):
|
||||
"""memory.go:781 — Latin/Cyrillic word-delimited script, or None for Han/kana/mixed."""
|
||||
script = None
|
||||
for ch in norm_key:
|
||||
if _is_cjk(ch):
|
||||
return None
|
||||
if _RE_LATIN.match(ch):
|
||||
if script == "cyr":
|
||||
return None
|
||||
script = "lat"
|
||||
elif _RE_CYRILLIC.match(ch):
|
||||
if script == "lat":
|
||||
return None
|
||||
script = "cyr"
|
||||
return script
|
||||
|
||||
def _suppress_unbounded_phonetic(self, ms, ntext):
|
||||
kept = []
|
||||
for (ki, s, e) in ms:
|
||||
script = self._spaced_phonetic_script(self.keys[ki])
|
||||
if script is not None:
|
||||
re_s = _RE_LATIN if script == "lat" else _RE_CYRILLIC
|
||||
before_bad = s > 0 and _is_letter(ntext[s - 1]) and re_s.match(ntext[s - 1])
|
||||
after_bad = e < len(ntext) and _is_letter(ntext[e]) and re_s.match(ntext[e])
|
||||
if before_bad or after_bad:
|
||||
continue
|
||||
kept.append((ki, s, e))
|
||||
return kept
|
||||
|
||||
def _match_trust(self, m, chapter: int):
|
||||
"""memory.go:656 — best disposition over spoiler-valid owners of the key; hasValid flag."""
|
||||
key = self.keys[m[0]]
|
||||
disp, has_valid = None, False
|
||||
for ei in self.key_owners[key]:
|
||||
e = self.entries[ei]
|
||||
if spoiler_blocked(e, chapter):
|
||||
continue
|
||||
d = disposition_for(e, key)
|
||||
if not has_valid or trust_rank(d) > trust_rank(disp):
|
||||
disp, has_valid = d, True
|
||||
return disp, has_valid
|
||||
|
||||
def _suppress_contained(self, ms, chapter: int):
|
||||
"""memory.go:684 — trust-gated longest-match. Returns (kept, trust_gate_events)."""
|
||||
kept = []
|
||||
gated = []
|
||||
for i, m in enumerate(ms):
|
||||
con_disp, con_valid = self._match_trust(m, chapter)
|
||||
contained = False
|
||||
refused = None
|
||||
for j, o in enumerate(ms):
|
||||
if i == j:
|
||||
continue
|
||||
# o must STRICTLY contain m (memory.go:696)
|
||||
if not (o[1] <= m[1] and o[2] >= m[2] and (o[2] - o[1]) > (m[2] - m[1])):
|
||||
continue
|
||||
sup_disp, sup_valid = self._match_trust(o, chapter)
|
||||
if not sup_valid:
|
||||
continue
|
||||
if (not con_valid) or trust_rank(sup_disp) >= trust_rank(con_disp):
|
||||
contained = True
|
||||
break
|
||||
if refused is None:
|
||||
refused = {
|
||||
"suppressor": self.keys[o[0]], "suppressor_disp": sup_disp,
|
||||
"protected": self.keys[m[0]], "protected_disp": con_disp,
|
||||
}
|
||||
if contained:
|
||||
continue
|
||||
kept.append(m)
|
||||
if refused is not None:
|
||||
gated.append(refused)
|
||||
return kept, _dedupe_trust_gate(gated)
|
||||
|
||||
def select(self, chunk: str, chapter: int, sticky_prev: dict, budget_tokens: int) -> Selection:
|
||||
ntext = normalize_source_key(chunk)
|
||||
occ = self._matches(ntext)
|
||||
occ = self._suppress_unbounded_phonetic(occ, ntext)
|
||||
occ, trust_gated = self._suppress_contained(occ, chapter)
|
||||
|
||||
# surviving key occurrences -> entries, longest firing key per entry (memory.go:340-348)
|
||||
matched_via = {}
|
||||
for m in occ:
|
||||
k = self.keys[m[0]]
|
||||
for ei in self.key_owners[k]:
|
||||
cur = matched_via.get(ei)
|
||||
if cur is None or len(k) > len(cur):
|
||||
matched_via[ei] = k
|
||||
|
||||
sel = Selection(trust_gated=trust_gated)
|
||||
hits = {} # id -> Picked (exact)
|
||||
for ei, e in enumerate(self.entries):
|
||||
via = matched_via.get(ei)
|
||||
if via is None:
|
||||
continue
|
||||
if spoiler_blocked(e, chapter):
|
||||
sel.rejected.append(Picked(e, via, REJECT))
|
||||
continue
|
||||
disp = disposition_for(e, via)
|
||||
hits[e.id] = Picked(e, via, disp)
|
||||
sel.active_ids[e.id] = disp
|
||||
|
||||
# sticky scene-inertia (memory.go:375-388) — inherit prior disposition
|
||||
for e in self.entries:
|
||||
if e.id not in sticky_prev:
|
||||
continue
|
||||
if e.id in hits:
|
||||
continue
|
||||
if spoiler_blocked(e, chapter):
|
||||
continue
|
||||
hits[e.id] = Picked(e, "sticky", sticky_prev[e.id])
|
||||
|
||||
# priority order (frozen-entry order, then stable-sort by priorityRank) (memory.go:393-401)
|
||||
order = [hits[e.id] for e in self.entries if e.id in hits]
|
||||
order.sort(key=priority_rank) # Python sort is stable
|
||||
|
||||
if budget_tokens > 0:
|
||||
used, cut = 0, len(order)
|
||||
for i, p in enumerate(order):
|
||||
used += glossary_line_tokens(p.entry.src, p.entry.dst)
|
||||
if used > budget_tokens:
|
||||
cut = i
|
||||
break
|
||||
sel.injected = order[:cut]
|
||||
sel.evicted = order[cut:]
|
||||
else:
|
||||
sel.injected = order
|
||||
return sel
|
||||
|
||||
|
||||
def spoiler_blocked(e: MemoryEntry, chapter: int) -> bool:
|
||||
if e.since_ch > 0 and chapter < e.since_ch:
|
||||
return True
|
||||
if e.until_ch > 0 and chapter > e.until_ch:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def disposition_for(e: MemoryEntry, via: str) -> str:
|
||||
if via != "sticky" and not e.allow_short and collision_prone_key(via):
|
||||
return AMBIGUOUS
|
||||
if e.status == "approved":
|
||||
return CONFIRMED
|
||||
return AMBIGUOUS
|
||||
|
||||
|
||||
def priority_rank(p: Picked) -> int:
|
||||
rank = 0
|
||||
if p.disp != CONFIRMED:
|
||||
rank |= 1 << 2
|
||||
if p.via == "sticky":
|
||||
rank |= 1 << 1
|
||||
if p.entry.status != "approved":
|
||||
rank |= 1 << 0
|
||||
return rank
|
||||
|
||||
|
||||
def _dedupe_trust_gate(evs):
|
||||
if len(evs) < 2:
|
||||
return evs
|
||||
seen = set()
|
||||
out = []
|
||||
for e in evs:
|
||||
k = (e["suppressor"], e["protected"])
|
||||
if k in seen:
|
||||
continue
|
||||
seen.add(k)
|
||||
out.append(e)
|
||||
return out
|
||||
|
||||
|
||||
def render_glossary_block(injected) -> str:
|
||||
"""memory.go:477 — translator glossary block."""
|
||||
lines = []
|
||||
for p in injected:
|
||||
if (p.entry.dst or "").strip() == "":
|
||||
continue
|
||||
line = f"{p.entry.src} → {p.entry.dst}"
|
||||
if p.disp != CONFIRMED:
|
||||
line += " ⟨проверить⟩"
|
||||
lines.append(line)
|
||||
if not lines:
|
||||
return ""
|
||||
return GLOSSARY_HEADER + "\n" + "\n".join(lines)
|
||||
|
||||
|
||||
def render_editor_constraint_block(injected) -> str:
|
||||
"""memory.go:515 — editor constraint block (CONFIRMED-only, dedup by dst)."""
|
||||
lines = []
|
||||
seen = set()
|
||||
for p in injected:
|
||||
if p.disp != CONFIRMED:
|
||||
continue
|
||||
dst = (p.entry.dst or "").strip()
|
||||
if dst == "" or dst in seen:
|
||||
continue
|
||||
seen.add(dst)
|
||||
lines.append(f"- «{dst}»")
|
||||
if not lines:
|
||||
return ""
|
||||
return EDITOR_HEADER + "\n" + "\n".join(lines)
|
||||
|
||||
|
||||
# --- seed loading (mirror loadGlossarySeed enough for the hot path) ---------------------------------
|
||||
def load_seed_rows(seed_path: str) -> list:
|
||||
d = yaml.safe_load(Path(seed_path).read_text(encoding="utf-8"))
|
||||
return d["terms"]
|
||||
|
||||
|
||||
def union_sticky(active_ids_per_chunk: list, k: int, depth: int = STICKY_DEPTH) -> dict:
|
||||
"""sticky_prev for chunk k = union of the previous `depth` chunks' active_ids (same chapter).
|
||||
active_ids_per_chunk[i] is the dict from chunk i's Selection.active_ids. (bookrun.go unionSticky)."""
|
||||
merged = {}
|
||||
for i in range(max(0, k - depth), k):
|
||||
for cid, disp in active_ids_per_chunk[i].items():
|
||||
merged[cid] = disp # later chunk wins on the disposition it carried
|
||||
return merged
|
||||
|
||||
|
||||
def build_bank(seed_path: str) -> MemoryBank:
|
||||
return MemoryBank(load_seed_rows(seed_path))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if "--selftest" in sys.argv:
|
||||
seed = "/home/ubuntu/books/gu-zhenren/guzhenren-seed-v2.yaml"
|
||||
bank = build_bank(seed)
|
||||
print(f"entries={len(bank.entries)} unique_keys={len(bank.keys)}")
|
||||
# normalize sanity: NFKC folds fullwidth space U+3000 -> U+0020 (kept; Go does NOT trim)
|
||||
assert normalize_source_key(" 方源 ") == " 方源 ", repr(normalize_source_key(" 方源 "))
|
||||
assert normalize_source_key("Ｑ版") == "q版", repr(normalize_source_key("Ｑ版"))
|
||||
# key "方源" still substring-matches inside the space-padded normalized text
|
||||
assert "方源" in normalize_source_key(" 方源 ")
|
||||
# a chunk from S2' head
|
||||
chunk = ("方源笑了，握紧拳头。古月方源！蛊师有九转。四代族长古月博。"
|
||||
"沈翠端着酒菜。方正是甲等资质，方源只是丙等。")
|
||||
sel = bank.select(chunk, chapter=1, sticky_prev={}, budget_tokens=800)
|
||||
print("\n-- exact active_ids (no sticky) --")
|
||||
for cid, disp in sel.active_ids.items():
|
||||
print(f" {disp:9} {cid.split(chr(31))[0]}")
|
||||
print("\n-- trust_gated events --")
|
||||
for g in sel.trust_gated:
|
||||
print(f" {g['suppressor']}({g['suppressor_disp']}) X-> {g['protected']}({g['protected_disp']})")
|
||||
print("\n-- TRANSLATOR block --")
|
||||
print(render_glossary_block(sel.injected))
|
||||
print("\n-- EDITOR block --")
|
||||
print(render_editor_constraint_block(sel.injected))
|
||||
# longest-match check: 古月方源 should suppress nested 方源 and 古月 as firing keys,
|
||||
# but the ENTITY 方源 still fires via its own occurrences elsewhere in the chunk.
|
||||
print("\nselftest OK")
|
||||
105
eval/exp15/preflight_models.py
Normal file
105
eval/exp15/preflight_models.py
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 preflight — live /models listing fact-check (two-directions rule, $0).
|
||||
|
||||
Lists /models for every vendor whose key exists and flags the arm/judge/miner
|
||||
slugs exp15 intends to use. LISTING ONLY — no generation, no billing. The alias
|
||||
caveat still holds (00-provider-quirks: "not in /models" != "doesn't work"), so a
|
||||
missing slug here is a flag to probe live, not a verdict.
|
||||
|
||||
Never prints keys. Persists raw listing to data for provenance (eval rule 1).
|
||||
Run: eval/.venv/bin/python eval/exp15/preflight_models.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# eval/.env holds the keys; scripts read it themselves (never printed).
|
||||
EVAL_DIR = Path(__file__).resolve().parent.parent
|
||||
load_dotenv(EVAL_DIR / ".env")
|
||||
|
||||
OUT_DIR = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
RAW = OUT_DIR / "preflight_models.json"
|
||||
|
||||
# (name, base_url, env_key, [target substrings to flag])
|
||||
PROVIDERS = [
|
||||
("deepseek", "https://api.deepseek.com/v1", "DEEPSEEK_API_KEY",
|
||||
["deepseek-v4-flash", "deepseek-v4-pro", "deepseek-chat", "deepseek-reasoner"]),
|
||||
("zai", "https://api.z.ai/api/paas/v4", "ZAI_API_KEY",
|
||||
["glm-5", "glm-5.1", "glm-5.2", "glm-4.7", "glm-4.6", "glm-4.5-air"]),
|
||||
("xai", "https://api.x.ai/v1", "XAI_API_KEY",
|
||||
["grok-4.3", "grok-4"]),
|
||||
("gemini", "https://generativelanguage.googleapis.com/v1beta/openai", "GEMINI_API_KEY",
|
||||
["gemini-3.1-pro", "gemini-3.5", "gemini-3.1-flash", "gemini-2.5"]),
|
||||
("openai", "https://api.openai.com/v1", "OPENAI_API_KEY",
|
||||
["gpt-5.4", "gpt-5-mini", "gpt-5"]),
|
||||
("mistral", "https://api.mistral.ai/v1", "MISTRAL_API_KEY",
|
||||
["mistral-large", "mistral-medium", "mistral-small"]),
|
||||
("kimi", "https://api.moonshot.ai/v1", "KIMI_API_KEY",
|
||||
["kimi-k2.6", "kimi-k2.5", "kimi-k2.7", "kimi-k2"]),
|
||||
]
|
||||
|
||||
|
||||
def list_models(name, base, env_key):
|
||||
key = os.environ.get(env_key)
|
||||
if not key:
|
||||
return {"provider": name, "error": f"no key in env ({env_key})"}
|
||||
url = base.rstrip("/") + "/models"
|
||||
try:
|
||||
r = requests.get(url, headers={"Authorization": f"Bearer {key}"}, timeout=40)
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"provider": name, "error": f"request failed: {type(e).__name__}: {e}"}
|
||||
out = {"provider": name, "status": r.status_code, "url": url}
|
||||
if r.status_code != 200:
|
||||
out["body_snippet"] = r.text[:300]
|
||||
return out
|
||||
try:
|
||||
data = r.json()
|
||||
except Exception: # noqa: BLE001
|
||||
out["error"] = "non-json body"
|
||||
out["body_snippet"] = r.text[:300]
|
||||
return out
|
||||
items = data.get("data", data if isinstance(data, list) else [])
|
||||
ids = sorted({(m.get("id") or m.get("name") or "") for m in items if isinstance(m, dict)})
|
||||
out["count"] = len(ids)
|
||||
out["model_ids"] = ids
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
results = []
|
||||
for name, base, env_key, targets in PROVIDERS:
|
||||
res = list_models(name, base, env_key)
|
||||
res["targets"] = targets
|
||||
if "model_ids" in res:
|
||||
ids = res["model_ids"]
|
||||
res["target_hits"] = {t: [m for m in ids if t in m] for t in targets}
|
||||
results.append(res)
|
||||
time.sleep(0.3)
|
||||
|
||||
RAW.write_text(json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
print(f"# exp15 preflight /models (raw -> {RAW})\n")
|
||||
for res in results:
|
||||
name = res["provider"]
|
||||
if "error" in res:
|
||||
print(f"[{name}] ERROR: {res['error']}")
|
||||
continue
|
||||
if res.get("status") != 200:
|
||||
print(f"[{name}] HTTP {res['status']}: {res.get('body_snippet','')[:120]}")
|
||||
continue
|
||||
print(f"[{name}] HTTP 200, {res['count']} models")
|
||||
for t, hits in res["target_hits"].items():
|
||||
mark = "OK " if hits else "-- "
|
||||
print(f" {mark}{t:<22} -> {hits if hits else 'NOT in /models (probe live before verdict)'}")
|
||||
print("\nNOTE: alias caveat (00-provider-quirks calendar): a slug absent here may still serve (HTTP 200).")
|
||||
print("Verdicts on slugs require BOTH this listing AND the vendor-doc workflow, per two-directions rule.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
100
eval/exp15/q0_reanalysis.py
Normal file
100
eval/exp15/q0_reanalysis.py
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — Q0 RE-ANALYSIS from existing judge_results_Q0.json (no re-run; same flat-chunking-both-sides
|
||||
reasoning as Q3a). Applies the orchestrator's corrections:
|
||||
|
||||
- TOST: the +0.041 point estimate is correct, but equivalence is NOT established. 90% CI upper bound
|
||||
exceeds the margin M=0.05 -> by the freeze rule the verdict is "insufficient data", and the §D5
|
||||
branch "editor mandates revised" does NOT trigger. (a0 = do-nothing DRAFT, arm = A0 editor FINAL;
|
||||
paired_diff = editor - donothing, positive = editor helps.)
|
||||
- SYMMETRY: the editor CUTS catastrophe votes (donothing -> editor), an effect ~the same size as the
|
||||
Q3a carryover "plus" — must sit in the headline next to it.
|
||||
- UNITS: catastrophe counts are judge VOTES, not traps. Report the distinct-trap counts and which traps
|
||||
the editor added flags on vs cleared.
|
||||
|
||||
Run: python q0_reanalysis.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from scipy import stats
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
PRIMARY = ("T-ana", "T-ell")
|
||||
M = 0.05 # TOST equivalence margin (freeze §1.5)
|
||||
|
||||
|
||||
def ci90(xs):
|
||||
xs = np.asarray(xs, float); n = len(xs)
|
||||
m = xs.mean(); sd = xs.std(ddof=1); se = sd / math.sqrt(n)
|
||||
t = stats.t.ppf(0.95, n - 1)
|
||||
return round(float(m), 4), round(float(sd), 4), round(float(m - t * se), 4), round(float(m + t * se), 4), n
|
||||
|
||||
|
||||
def tost(xs, margin):
|
||||
"""Two one-sided tests for equivalence within +-margin. Equivalent iff BOTH p<0.05 (== 90% CI inside
|
||||
(-margin, +margin))."""
|
||||
xs = np.asarray(xs, float); n = len(xs)
|
||||
m = xs.mean(); se = xs.std(ddof=1) / math.sqrt(n)
|
||||
if se == 0:
|
||||
return None
|
||||
t_lo = (m - (-margin)) / se # H0: mu <= -margin
|
||||
t_hi = (m - margin) / se # H0: mu >= +margin
|
||||
p_lo = 1 - stats.t.cdf(t_lo, n - 1)
|
||||
p_hi = stats.t.cdf(t_hi, n - 1)
|
||||
return {"p_lower": round(float(p_lo), 4), "p_upper": round(float(p_hi), 4),
|
||||
"equivalent_90": bool(p_lo < 0.05 and p_hi < 0.05)}
|
||||
|
||||
|
||||
def main():
|
||||
d = json.load(open(SD / "judge_results_Q0.json"))
|
||||
res = d["results"] if isinstance(d, dict) and "results" in d else d
|
||||
allr = res
|
||||
prim = [r for r in res if r["type"] in PRIMARY]
|
||||
|
||||
out = {}
|
||||
for label, rows in (("all", allr), ("primary", prim), ("T-ana", [r for r in res if r["type"] == "T-ana"])):
|
||||
pd = [r["paired_diff_arm_minus_a0"] for r in rows if r["paired_diff_arm_minus_a0"] is not None]
|
||||
m, sd, lo, hi, n = ci90(pd)
|
||||
t = tost(pd, M)
|
||||
# catastrophe: arm = editor(final), a0 = donothing(draft)
|
||||
cat_editor = sum(r["catastrophe"]["arm"] for r in rows)
|
||||
cat_donothing = sum(r["catastrophe"]["a0"] for r in rows)
|
||||
out[label] = {"editor_minus_donothing_mean": m, "sd": sd, "CI90": [lo, hi], "n": n,
|
||||
"TOST_M": M, "TOST": t,
|
||||
"equivalence_established": bool(t and t["equivalent_90"]),
|
||||
"cat_votes_editor_vs_donothing": [cat_editor, cat_donothing],
|
||||
"cat_reduction_pct": round(100 * (cat_donothing - cat_editor) / cat_donothing, 1) if cat_donothing else None}
|
||||
verdict = ("EQUIVALENT (CI within +-M)" if (t and t["equivalent_90"])
|
||||
else f"NOT established (CI upper {hi} exceeds M={M}) -> insufficient data, D5 branch NOT triggered")
|
||||
print(f"[{label}] editor-donothing {m} sd {sd} CI90 [{lo},{hi}] n={n}")
|
||||
print(f" TOST {t} -> {verdict}")
|
||||
print(f" catastrophe VOTES editor {cat_editor} vs donothing {cat_donothing} "
|
||||
f"(editor cuts {out[label]['cat_reduction_pct']}%)")
|
||||
|
||||
# UNITS: distinct traps with catastrophe votes (not the same as vote counts); who added/cleared flags
|
||||
added, cleared, net = [], [], []
|
||||
for r in prim:
|
||||
ce, cd = r["catastrophe"]["arm"], r["catastrophe"]["a0"] # editor, donothing
|
||||
if ce > cd:
|
||||
added.append((r["trap"], cd, ce))
|
||||
elif ce < cd:
|
||||
cleared.append((r["trap"], cd, ce))
|
||||
trap_editor = sorted({r["trap"] for r in prim if r["catastrophe"]["arm"] > 0})
|
||||
trap_donothing = sorted({r["trap"] for r in prim if r["catastrophe"]["a0"] > 0})
|
||||
out["units"] = {"distinct_traps_with_editor_cat": len(trap_editor),
|
||||
"distinct_traps_with_donothing_cat": len(trap_donothing),
|
||||
"editor_ADDED_flags_on": added, "editor_CLEARED_flags_on": cleared}
|
||||
print(f"\n[units] distinct traps w/ catastrophe: editor {len(trap_editor)} vs donothing {len(trap_donothing)} "
|
||||
f"(votes != traps)")
|
||||
print(f" editor ADDED flags on {len(added)} traps: {added}")
|
||||
print(f" editor CLEARED flags on {len(cleared)} traps: {cleared}")
|
||||
|
||||
(SD / "q0_reanalysis.json").write_text(json.dumps(out, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\n-> {SD/'q0_reanalysis.json'}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
121
eval/exp15/q1_setup.py
Normal file
121
eval/exp15/q1_setup.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — Q1 arm setup + size-match + cut-alignment (checkpoint item e; design §D2/§1.14).
|
||||
|
||||
A0 = greedy-1500 (the validated greedy port reproduces A0's 24 chunks byte-identically). The LOGICAL arm
|
||||
is the §B1.5 DP chunker sized to A0's ACTUAL mean est_out (±10%), so the ONLY Q1 variable is cut
|
||||
ALIGNMENT, not size (size = Q2's domain). Reports:
|
||||
- est_out distributions (greedy vs logical) + size-match verdict (±10% mean)
|
||||
- cut-alignment (secondary Q1 metric): fraction of each arm's cuts landing ON scene-shift seams
|
||||
(higher=better) vs continuation/negative seams (lower=better), and fraction of seams hit.
|
||||
This is the honest "lever engaged / Q1 decidable a priori" evidence.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from chunker import greedy_chunks, logical_chunks, est_out, normalize_source
|
||||
|
||||
S2 = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime.txt")
|
||||
SEAMS = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime_seams.json")
|
||||
LABELS = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime_seam_labels.json")
|
||||
OUT = Path("/home/ubuntu/books/gu-zhenren/exp15/q1_setup.json")
|
||||
TOL = 0 # a cut "hits" a seam iff exact rune offset match (both are line/sentence boundaries)
|
||||
|
||||
|
||||
def greedy_cut_offsets(raw):
|
||||
"""Rune offsets (in NORMALIZED text) where each greedy chunk starts (excl 0)."""
|
||||
chs = greedy_chunks(raw)
|
||||
offs, cur = [], 0
|
||||
for c in chs:
|
||||
if cur > 0:
|
||||
offs.append(cur)
|
||||
cur += len(c)
|
||||
return chs, offs
|
||||
|
||||
|
||||
def main():
|
||||
raw = S2.read_text(encoding="utf-8")
|
||||
norm = normalize_source(raw)
|
||||
# NOTE: greedy_chunks re-joins paragraphs with "\n\n" and trims; its cut offsets are in the
|
||||
# concatenated-chunk space. For seam alignment we need offsets in the SAME space as the seams
|
||||
# (the normalized S2'). We recompute cut offsets by locating each chunk's head in `norm`.
|
||||
g_chunks = greedy_chunks(raw)
|
||||
g_offs = _locate_starts(g_chunks, norm)
|
||||
g_out = [est_out(c) for c in g_chunks]
|
||||
mean_g = sum(g_out) / len(g_out)
|
||||
|
||||
# logical arm sized to A0 mean est_out
|
||||
plan = logical_chunks(norm, target_out=mean_g, hard_max_out=max(g_out))
|
||||
l_chunks = [c[0] for c in plan.chunks]
|
||||
l_offs = _locate_starts(l_chunks, norm)
|
||||
l_out = [c[1] for c in plan.chunks]
|
||||
mean_l = sum(l_out) / len(l_out)
|
||||
|
||||
# seams (already aligned to normalized/current S2'? seams were computed on the LF text = S2'.txt;
|
||||
# norm = S2'.txt stripped of leading/trailing ws. leading strip shifts offsets by the stripped prefix.)
|
||||
strip_shift = raw.find(norm[:10]) # how many runes were stripped from the front
|
||||
seams = json.load(open(SEAMS))["seams"]
|
||||
labels = json.load(open(LABELS))["labels"]
|
||||
seam_pts = [{"id": s["seam_id"], "off": s["rune_offset"] - strip_shift,
|
||||
"class": labels[str(s["seam_id"])]["class"]} for s in seams]
|
||||
|
||||
def hits(offs):
|
||||
offset_set = set(offs)
|
||||
res = {"scene-shift": 0, "continuation": 0, "seams_hit": []}
|
||||
for sp in seam_pts:
|
||||
hit = any(abs(o - sp["off"]) <= TOL for o in offs)
|
||||
if hit:
|
||||
res[sp["class"]] += 1
|
||||
res["seams_hit"].append(sp["id"])
|
||||
return res
|
||||
|
||||
g_hits, l_hits = hits(g_offs), hits(l_offs)
|
||||
n_scene = sum(1 for s in seam_pts if s["class"] == "scene-shift")
|
||||
n_cont = sum(1 for s in seam_pts if s["class"] == "continuation")
|
||||
|
||||
lo, hi = mean_g * 0.9, mean_g * 1.1
|
||||
result = {
|
||||
"greedy_A0": {"n_chunks": len(g_chunks), "n_cuts": len(g_offs), "mean_est_out": round(mean_g, 1),
|
||||
"min": round(min(g_out), 1), "max": round(max(g_out), 1), "cut_offsets": g_offs},
|
||||
"logical": {"n_chunks": len(l_chunks), "n_cuts": len(l_offs), "mean_est_out": round(mean_l, 1),
|
||||
"min": round(min(l_out), 1), "max": round(max(l_out), 1), "cut_offsets": l_offs,
|
||||
"right_boundary_classes": [c[2] for c in plan.chunks[:-1]]},
|
||||
"size_match": {"target": round(mean_g, 1), "logical_mean": round(mean_l, 1),
|
||||
"within_10pct": lo <= mean_l <= hi, "band": [round(lo, 1), round(hi, 1)]},
|
||||
"seams": {"n_scene_shift": n_scene, "n_continuation": n_cont},
|
||||
"cut_alignment": {
|
||||
"greedy": {"scene_shift_hits": g_hits["scene-shift"], "continuation_hits": g_hits["continuation"],
|
||||
"seams_hit": g_hits["seams_hit"]},
|
||||
"logical": {"scene_shift_hits": l_hits["scene-shift"], "continuation_hits": l_hits["continuation"],
|
||||
"seams_hit": l_hits["seams_hit"]},
|
||||
"interpretation": "logical should hit MORE scene-shift seams and FEWER continuation seams than greedy",
|
||||
},
|
||||
}
|
||||
OUT.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"greedy A0: {len(g_chunks)} chunks, mean est_out {mean_g:.0f}")
|
||||
print(f"logical: {len(l_chunks)} chunks, mean est_out {mean_l:.0f} size-match(±10%)={result['size_match']['within_10pct']}")
|
||||
print(f"\nseams: {n_scene} scene-shift, {n_cont} continuation")
|
||||
print(f"cut-alignment (exact rune hits):")
|
||||
print(f" greedy : scene-shift {g_hits['scene-shift']}/{n_scene} continuation {g_hits['continuation']}/{n_cont} seams_hit={g_hits['seams_hit']}")
|
||||
print(f" logical: scene-shift {l_hits['scene-shift']}/{n_scene} continuation {l_hits['continuation']}/{n_cont} seams_hit={l_hits['seams_hit']}")
|
||||
print(f"-> {OUT}")
|
||||
|
||||
|
||||
def _locate_starts(chunks, norm):
|
||||
"""Rune offset in `norm` where each chunk (except the first) starts, via its head."""
|
||||
offs, pos = [], 0
|
||||
for i, c in enumerate(chunks):
|
||||
head = c[:25]
|
||||
idx = norm.find(head, max(0, pos - 5))
|
||||
if idx < 0:
|
||||
idx = norm.find(head)
|
||||
if i > 0 and idx >= 0:
|
||||
offs.append(idx)
|
||||
pos = idx + len(c) if idx >= 0 else pos + len(c)
|
||||
return offs
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
123
eval/exp15/q1b_reanalysis.py
Normal file
123
eval/exp15/q1b_reanalysis.py
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — Q1b (full-chunk re-judge) + FLOOR analysis. Reads the fresh full-chunk judge_results_Q1b.json
|
||||
and judge_results_FLOOR.json and produces the corrected, floor-relative Q1b verdict.
|
||||
|
||||
Key checks (the whole point of the re-judge):
|
||||
1. CORRECTED effect: oracle-minus-flat paired diff, primary (T-ana+T-ell) and per type, with 90% CI, LOO,
|
||||
catastrophes — vs the v1 head-excerpt headline (-0.564) to show the artifact's magnitude.
|
||||
2. ARTIFACT-GONE check: split by oracle_frac (dep position in the oracle chunk). Under the old 1100-char
|
||||
head-excerpt, dep-in-window cells were +0.238 and dep-out were -0.872 (the truncation signature). With
|
||||
FULL chunks the split MUST collapse (both similar) — if dep-out is still strongly negative the fix failed.
|
||||
3. FLOOR-relative: is |Q1b effect| within the A0-vs-A0' full-chunk noise floor? Effects under the floor
|
||||
are not interpretable (freeze §1.5).
|
||||
4. Per-vote audit available (judges.py now persists votes) — catastrophe counts reconstructable.
|
||||
|
||||
Run (after both blocks): python q1b_reanalysis.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from scipy import stats
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
PRIMARY = ("T-ana", "T-ell")
|
||||
|
||||
|
||||
def ci90(xs):
|
||||
xs = np.asarray(xs, float); n = len(xs)
|
||||
if n < 2:
|
||||
return (round(float(xs.mean()), 4) if n else 0.0, None, None, n)
|
||||
m = xs.mean(); se = xs.std(ddof=1) / math.sqrt(n); t = stats.t.ppf(0.95, n - 1)
|
||||
return round(float(m), 4), round(float(m - t * se), 4), round(float(m + t * se), 4), n
|
||||
|
||||
|
||||
def load(name):
|
||||
p = SD / name
|
||||
if not p.exists():
|
||||
return None
|
||||
d = json.load(open(p))
|
||||
return d["results"] if isinstance(d, dict) and "results" in d else d
|
||||
|
||||
|
||||
def block_stats(rows, label):
|
||||
pd = [r["paired_diff_arm_minus_a0"] for r in rows if r.get("paired_diff_arm_minus_a0") is not None]
|
||||
m, lo, hi, n = ci90(pd)
|
||||
cat_arm = sum(r["catastrophe"]["arm"] for r in rows)
|
||||
cat_a0 = sum(r["catastrophe"]["a0"] for r in rows)
|
||||
loo = {"without_grok-4.3": [], "without_mistral-large-latest": []}
|
||||
for r in rows:
|
||||
for k, v in (r.get("leave_one_judge_out") or {}).items():
|
||||
if k in loo:
|
||||
loo[k].append(v)
|
||||
loo_m = {k: round(float(np.mean(v)), 4) for k, v in loo.items() if v}
|
||||
trunc = sum(1 for r in rows if r.get("budget_truncated"))
|
||||
return {"label": label, "mean": m, "CI90": [lo, hi], "n": n,
|
||||
"catastrophes_arm_vs_a0": [cat_arm, cat_a0], "loo_mean": loo_m, "budget_truncated_cells": trunc}
|
||||
|
||||
|
||||
def main():
|
||||
q1b = load("judge_results_Q1b.json")
|
||||
floor = load("judge_results_FLOOR.json")
|
||||
v1 = load("judge_results_Q1b_v1_headexcerpt.json")
|
||||
out = {}
|
||||
|
||||
if q1b is None:
|
||||
print("Q1b results not present yet."); return
|
||||
prim = [r for r in q1b if r["type"] in PRIMARY]
|
||||
tana = [r for r in q1b if r["type"] == "T-ana"]
|
||||
tent = [r for r in q1b if r["type"] == "T-ent"]
|
||||
out["Q1b_fullchunk"] = {"primary": block_stats(prim, "primary(T-ana+T-ell)"),
|
||||
"T-ana": block_stats(tana, "T-ana"),
|
||||
"T-ent_control": block_stats(tent, "T-ent control") if tent else None}
|
||||
print("=== Q1b FULL-CHUNK (oracle - flat; + = oracle better) ===")
|
||||
for k in ("primary", "T-ana", "T-ent_control"):
|
||||
s = out["Q1b_fullchunk"][k]
|
||||
if s:
|
||||
print(f" {s['label']:22} mean {s['mean']:+.4f} CI90 {s['CI90']} n={s['n']} "
|
||||
f"cat oracle/flat {s['catastrophes_arm_vs_a0']} LOO {s['loo_mean']} trunc={s['budget_truncated_cells']}")
|
||||
|
||||
# artifact-gone check: split by oracle_frac
|
||||
din = [r for r in prim if (r.get("oracle_frac") or 0) < 0.5]
|
||||
dout = [r for r in prim if (r.get("oracle_frac") or 0) >= 0.5]
|
||||
si, so = ci90([r["paired_diff_arm_minus_a0"] for r in din]), ci90([r["paired_diff_arm_minus_a0"] for r in dout])
|
||||
out["artifact_gone_check"] = {"dep_in_window_frac<0.5": {"mean": si[0], "n": si[3]},
|
||||
"dep_out_window_frac>=0.5": {"mean": so[0], "n": so[3]},
|
||||
"v1_reference": {"dep_in": 0.238, "dep_out": -0.872},
|
||||
"collapsed": abs(si[0] - so[0]) < 0.3}
|
||||
print(f"\n=== ARTIFACT-GONE check (full chunks should collapse the split) ===")
|
||||
print(f" dep-in (ofrac<0.5): mean {si[0]:+.4f} n={si[3]} (v1 head-excerpt: +0.238)")
|
||||
print(f" dep-out (ofrac>=0.5): mean {so[0]:+.4f} n={so[3]} (v1 head-excerpt: -0.872)")
|
||||
print(f" -> split {'COLLAPSED (artifact gone)' if out['artifact_gone_check']['collapsed'] else 'STILL PRESENT — investigate'}")
|
||||
|
||||
if v1:
|
||||
vp = [r for r in v1 if r["type"] == "T-ana"]
|
||||
vm, *_ = ci90([r["paired_diff_arm_minus_a0"] for r in vp])
|
||||
out["v1_vs_v2"] = {"v1_Tana_headexcerpt": vm, "v2_Tana_fullchunk": out["Q1b_fullchunk"]["T-ana"]["mean"],
|
||||
"artifact_magnitude": round(out["Q1b_fullchunk"]["T-ana"]["mean"] - vm, 4)}
|
||||
print(f"\n=== v1 (head-excerpt) vs v2 (full-chunk) T-ana ===")
|
||||
print(f" v1 {vm:+.4f} -> v2 {out['Q1b_fullchunk']['T-ana']['mean']:+.4f} "
|
||||
f"(the artifact moved the headline by {out['v1_vs_v2']['artifact_magnitude']:+.4f})")
|
||||
|
||||
if floor is not None:
|
||||
fpd = [abs(r["paired_diff_arm_minus_a0"]) for r in floor if r.get("paired_diff_arm_minus_a0") is not None]
|
||||
fmean = round(float(np.mean(fpd)), 4); fmax = round(float(np.max(fpd)), 4)
|
||||
fsigned = ci90([r["paired_diff_arm_minus_a0"] for r in floor if r.get("paired_diff_arm_minus_a0") is not None])
|
||||
out["FLOOR_fullchunk"] = {"mean_abs_diff": fmean, "max_abs_diff": fmax,
|
||||
"signed_mean": fsigned[0], "signed_CI90": [fsigned[1], fsigned[2]], "n": fsigned[3]}
|
||||
prim_abs = abs(out["Q1b_fullchunk"]["primary"]["mean"])
|
||||
out["FLOOR_fullchunk"]["Q1b_primary_exceeds_floor"] = bool(prim_abs > fmean)
|
||||
print(f"\n=== FLOOR (A0 vs A0', full-chunk) ===")
|
||||
print(f" |diff| mean {fmean} max {fmax} signed {fsigned[0]:+.4f} CI90 [{fsigned[1]},{fsigned[2]}] n={fsigned[3]}")
|
||||
print(f" Q1b primary |{prim_abs:.4f}| {'EXCEEDS' if prim_abs>fmean else 'is WITHIN'} the floor {fmean}")
|
||||
else:
|
||||
print("\n(FLOOR results not present yet — run FLOOR block first.)")
|
||||
|
||||
(SD / "q1b_reanalysis.json").write_text(json.dumps(out, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\n-> {SD/'q1b_reanalysis.json'}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
137
eval/exp15/q3a_reanalysis.py
Normal file
137
eval/exp15/q3a_reanalysis.py
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — Q3a RE-ANALYSIS from existing judge_results_Q3a.json (no re-run: the head-excerpt artifact is
|
||||
Q1b-specific — Q3a's a0 and every arm share FLAT chunking, so the dependent sits at the head in BOTH
|
||||
sides -> no truncation asymmetry). Applies the orchestrator's corrections:
|
||||
|
||||
(b) primary = T-ana + T-ell (pre-reg §1.5), NOT the post-hoc T-ana-only "+0.042"; disclose T-ell-43.
|
||||
(c) BOTH estimators reported with 90% CI: paired (mean per-cell arm-a0) AND marginal arm-acc
|
||||
(mean arm_acc - mean a0_acc). They diverge; report both.
|
||||
(d) leave-one-judge-out sign per mode (src stable? target-side/final flip?).
|
||||
(e) POOLED fix-rate: A0-failed traps defined ONCE by pooling a0 across modes (the per-mode denominators
|
||||
1/4,1/4,0/3 were an artifact of re-judging A0 separately per mode). fix = arm resolves it.
|
||||
(a) a0<->a0 NOISE PROXY: the same flat-a0 chunk is judged 3x (once per mode-cell) -> its a0_accuracy
|
||||
spread across modes is pure judge noise. Reported as a preliminary floor; the dedicated FLOOR block
|
||||
(A0 vs A0', judge_run.py --run FLOOR) is authoritative.
|
||||
|
||||
Run: python q3a_reanalysis.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import math
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from scipy import stats
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
PRIMARY = ("T-ana", "T-ell")
|
||||
MODES = ("q3a-src", "q3a-draft", "q3a-final")
|
||||
FAIL = 0.5 # a0_accuracy < FAIL == "A0 failed this trap" (correct=1/partial=.5/wrong=0)
|
||||
|
||||
|
||||
def ci90(xs):
|
||||
xs = np.asarray(xs, float)
|
||||
n = len(xs)
|
||||
if n < 2:
|
||||
return (float(xs.mean()) if n else 0.0, float("nan"), float("nan"), n)
|
||||
m = xs.mean(); se = xs.std(ddof=1) / math.sqrt(n)
|
||||
t = stats.t.ppf(0.95, n - 1)
|
||||
return round(m, 4), round(m - t * se, 4), round(m + t * se, 4), n
|
||||
|
||||
|
||||
def paired_p(xs):
|
||||
xs = np.asarray(xs, float)
|
||||
if len(xs) < 2 or xs.std() == 0:
|
||||
return None
|
||||
return round(float(stats.ttest_1samp(xs, 0.0).pvalue), 3)
|
||||
|
||||
|
||||
def main():
|
||||
d = json.load(open(SD / "judge_results_Q3a.json"))
|
||||
res = d["results"] if isinstance(d, dict) and "results" in d else d
|
||||
by_mode = defaultdict(list)
|
||||
for r in res:
|
||||
by_mode[r["arm"]].append(r)
|
||||
|
||||
# a0<->a0 noise proxy: spread of a0_accuracy for the same trap across its mode-cells (identical a0 text)
|
||||
a0_by_trap = defaultdict(dict)
|
||||
for r in res:
|
||||
if r["type"] in PRIMARY:
|
||||
a0_by_trap[r["trap"]][r["arm"]] = r["a0_accuracy"]
|
||||
noises = []
|
||||
for trap, mm in a0_by_trap.items():
|
||||
vals = [v for v in mm.values() if v is not None]
|
||||
if len(vals) >= 2:
|
||||
noises.append(max(vals) - min(vals))
|
||||
floor_proxy = {"mean_a0_spread": round(float(np.mean(noises)), 4),
|
||||
"max_a0_spread": round(float(np.max(noises)), 4), "n_traps": len(noises)}
|
||||
|
||||
out = {"floor_proxy_a0_vs_a0": floor_proxy, "modes": {}}
|
||||
print(f"[a0<->a0 noise proxy] mean spread {floor_proxy['mean_a0_spread']} "
|
||||
f"max {floor_proxy['max_a0_spread']} (dedicated FLOOR block is authoritative)")
|
||||
|
||||
for mode in MODES:
|
||||
rows = by_mode.get(mode, [])
|
||||
prim = [r for r in rows if r["type"] in PRIMARY]
|
||||
tana = [r for r in rows if r["type"] == "T-ana"]
|
||||
# paired estimator (per-cell arm-a0)
|
||||
pd_prim = [r["paired_diff_arm_minus_a0"] for r in prim if r["paired_diff_arm_minus_a0"] is not None]
|
||||
pd_tana = [r["paired_diff_arm_minus_a0"] for r in tana if r["paired_diff_arm_minus_a0"] is not None]
|
||||
# marginal arm-acc estimator (mean arm - mean a0)
|
||||
arm_prim = np.mean([r["arm_accuracy"] for r in prim])
|
||||
a0_prim = np.mean([r["a0_accuracy"] for r in prim])
|
||||
marg_prim = round(float(arm_prim - a0_prim), 4)
|
||||
# LOO sign (mean of per-cell LOO diffs, which drop one judge)
|
||||
loo = {"without_grok-4.3": [], "without_mistral-large-latest": []}
|
||||
for r in prim:
|
||||
for k, v in (r.get("leave_one_judge_out") or {}).items():
|
||||
if k in loo:
|
||||
loo[k].append(v)
|
||||
loo_means = {k: round(float(np.mean(v)), 4) for k, v in loo.items() if v}
|
||||
# catastrophes
|
||||
cat_arm = sum(r["catastrophe"]["arm"] for r in prim)
|
||||
cat_a0 = sum(r["catastrophe"]["a0"] for r in prim)
|
||||
pm, plo, phi, pn = ci90(pd_prim)
|
||||
tm, tlo, thi, tn = ci90(pd_tana)
|
||||
mde = round(2.8 * np.std(pd_prim, ddof=1) / math.sqrt(len(pd_prim)), 4) if len(pd_prim) > 1 else None
|
||||
out["modes"][mode] = {
|
||||
"primary_paired_mean": pm, "primary_paired_CI90": [plo, phi], "primary_n": pn,
|
||||
"primary_paired_p": paired_p(pd_prim), "primary_MDE90": mde,
|
||||
"primary_marginal_armacc": marg_prim,
|
||||
"tana_only_paired_mean": tm, "tana_only_CI90": [tlo, thi], "tana_n": tn,
|
||||
"loo_mean_diff": loo_means,
|
||||
"primary_catastrophes_arm_vs_a0": [cat_arm, cat_a0],
|
||||
}
|
||||
print(f"\n[{mode}]")
|
||||
print(f" primary(T-ana+T-ell) PAIRED {pm} CI90 [{plo},{phi}] n={pn} p={paired_p(pd_prim)} MDE90~{mde}")
|
||||
print(f" primary MARGINAL arm-acc {marg_prim} (metric-swap gap vs paired = {round(marg_prim-pm,4)})")
|
||||
print(f" T-ana-only PAIRED {tm} CI90 [{tlo},{thi}] n={tn} <- the reported headline")
|
||||
print(f" LOO mean diff {loo_means} catastrophes arm/a0 {cat_arm}/{cat_a0}")
|
||||
|
||||
# pooled fix-rate: A0-failed defined by pooling a0 across modes (mean a0 over the 3 cells < FAIL)
|
||||
pooled_a0 = {t: np.mean(list(v.values())) for t, v in a0_by_trap.items()}
|
||||
a0_failed = [t for t, a in pooled_a0.items() if a < FAIL]
|
||||
fixrate = {}
|
||||
for mode in MODES:
|
||||
rows = {r["trap"]: r for r in by_mode.get(mode, []) if r["type"] in PRIMARY}
|
||||
fixed = sum(1 for t in a0_failed if t in rows and rows[t]["arm_accuracy"] >= FAIL)
|
||||
fixrate[mode] = f"{fixed}/{len(a0_failed)} ({round(100*fixed/len(a0_failed)) if a0_failed else 0}%)"
|
||||
out["pooled_fix_rate"] = {"A0_failed_traps": sorted(a0_failed), "n_failed": len(a0_failed), "by_mode": fixrate}
|
||||
print(f"\n[pooled fix-rate] A0-failed (pooled a0<{FAIL}) = {len(a0_failed)} traps {sorted(a0_failed)}")
|
||||
for mode, fr in fixrate.items():
|
||||
print(f" {mode}: {fr}")
|
||||
|
||||
# T-ell-43 disclosure (winner-caused catastrophe)
|
||||
tell43 = {r["arm"]: {"a0_acc": r["a0_accuracy"], "arm_acc": r["arm_accuracy"],
|
||||
"cat_arm": r["catastrophe"]["arm"], "cat_a0": r["catastrophe"]["a0"]}
|
||||
for r in res if r["trap"] == "T-ell-43"}
|
||||
out["T-ell-43_disclosure"] = tell43
|
||||
print(f"\n[T-ell-43 disclosure] {json.dumps(tell43, ensure_ascii=False)}")
|
||||
|
||||
(SD / "q3a_reanalysis.json").write_text(json.dumps(out, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\n-> {SD/'q3a_reanalysis.json'}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
243
eval/exp15/reflow_omission.py
Normal file
243
eval/exp15/reflow_omission.py
Normal file
|
|
@ -0,0 +1,243 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — reflow-invariant omission metric + Q2b re-attribution ($0). Design §D3 + orchestrator amendments.
|
||||
|
||||
The metric asks, per SOURCE sentence-atom, whether the atom appears ANYWHERE in the target (position-
|
||||
independent = reflow-invariant, so legitimate merge/reorder is NOT penalized). Primary atom = glossary
|
||||
ENTITY (design-relevant: entities/terms = 62.5% doc-level errors, BWB).
|
||||
|
||||
ORCHESTRATOR AMENDMENTS (2026-07-17):
|
||||
(1) Entity match by LEMMA (pymorphy3), NOT membership in decl.forms. A form present but outside the
|
||||
pinned decl.forms = `inflection_gap` FLAG for span-check, NOT an omission (else the D30.9 §B5
|
||||
inflection_gap class fabricates ~7-8% false omissions).
|
||||
(2) Pre-registered variance-check: entity-omission ~0 across ALL sizes => report "INSTRUMENT CEILING",
|
||||
never "size is safe" (anti-false-negative; the D32.4 lesson).
|
||||
(3) Entity-free (holistic) content-drop detection runs ONLY if a crude content-length FLOOR backstop
|
||||
fires; otherwise complex drops are span-judge work (don't over-engineer).
|
||||
(4) Provenance to BYTE identity: source = records.json chapter_source join (== exp14_sizecurve.py:30-32,
|
||||
the exact bytes sizecurve translated). Asserted below.
|
||||
|
||||
VALIDATION (mandated): the metric is validated on a hand-proofread whole-chapter output BEFORE scoring
|
||||
arms — run with --validate <ch> to dump per-entity verdicts for manual read; only after that, --score.
|
||||
|
||||
Run: eval/.venv/bin/python eval/exp15/reflow_omission.py --validate 13 # dump for hand-proofread
|
||||
eval/.venv/bin/python eval/exp15/reflow_omission.py --score # Q2b table (after validation)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
import pymorphy3
|
||||
import yaml
|
||||
|
||||
BOOK = Path("/home/ubuntu/books/gu-zhenren")
|
||||
SEED = BOOK / "guzhenren-seed-v2.yaml"
|
||||
RECORDS = BOOK / "rerun" / "records.json"
|
||||
SIZE_DIR = BOOK / "exp14" / "sizecurve"
|
||||
OUT = BOOK / "exp15" / "q2b_omission.json"
|
||||
SIZES = ["800c", "1600c", "3200c", "whole"]
|
||||
CHAPTERS = [13, 17]
|
||||
CONTENT_FLOOR = 0.5 # amendment 3: target ru-char / fertility-expected < this => holistic-drop backstop fires
|
||||
|
||||
_MORPH = pymorphy3.MorphAnalyzer()
|
||||
CYR = re.compile(r"[А-Яа-яЁё]+")
|
||||
|
||||
|
||||
@lru_cache(maxsize=200_000)
|
||||
def lemma(tok: str) -> str:
|
||||
return _MORPH.parse(tok)[0].normal_form
|
||||
|
||||
|
||||
def ru_tokens(text: str):
|
||||
return CYR.findall(text)
|
||||
|
||||
|
||||
def target_index(text: str):
|
||||
"""surface tokens, lemma set, and lemma->surfaces map of a ru target."""
|
||||
surf = ru_tokens(text)
|
||||
lem_of = {s: lemma(s.lower()) for s in set(surf)}
|
||||
lemset = set(lem_of.values())
|
||||
lem2surf = {}
|
||||
for s in surf:
|
||||
lem2surf.setdefault(lem_of[s], set()).add(s)
|
||||
return {"lemset": lemset, "lem2surf": lem2surf}
|
||||
|
||||
|
||||
def load_seed_entities():
|
||||
d = yaml.safe_load(SEED.read_text(encoding="utf-8"))
|
||||
ents = []
|
||||
for t in d["terms"]:
|
||||
src = t.get("src")
|
||||
dst = (t.get("dst") or "").strip()
|
||||
if not src or not dst:
|
||||
continue
|
||||
srcs = [src] + [a.get("alias") for a in (t.get("aliases") or []) if a.get("alias")]
|
||||
forms = set()
|
||||
decl = t.get("decl") or {}
|
||||
for f in (decl.get("forms") or []):
|
||||
forms.add(f)
|
||||
forms.add(dst)
|
||||
# distinctive ru lemmas of the dst (content tokens len>=3)
|
||||
dtoks = ru_tokens(dst)
|
||||
content = [x for x in dtoks if len(x) >= 3]
|
||||
lemmas = [lemma(x.lower()) for x in (content or dtoks)]
|
||||
ents.append({"src_variants": srcs, "dst": dst, "type": t.get("type"),
|
||||
"decl_forms": forms, "lemmas": lemmas, "dst_tokens": dtoks})
|
||||
return ents
|
||||
|
||||
|
||||
def chapter_source(ch):
|
||||
"""BYTE-identical to exp14_sizecurve.py:30-32 (join of records.json chunk sources by '\\n')."""
|
||||
recs = {(r["chapter"], r["chunk_idx"]): r for r in json.load(open(RECORDS))}
|
||||
cks = sorted(c for (c0, c) in recs if c0 == ch)
|
||||
return "\n".join(recs[(ch, c)]["source"] for c in cks)
|
||||
|
||||
|
||||
def score(source_text: str, target_text: str, ents):
|
||||
"""Return per-entity verdicts + summary for one (source, target) pair. Reflow-invariant."""
|
||||
tgt = target_index(target_text)
|
||||
present_src = [e for e in ents if any(s in source_text for s in e["src_variants"])]
|
||||
results = []
|
||||
for e in present_src:
|
||||
elem = [l for l in e["lemmas"]]
|
||||
hit = [l for l in elem if l in tgt["lemset"]]
|
||||
if not elem:
|
||||
verdict = "skip"
|
||||
elif len(hit) == len(elem):
|
||||
verdict = "present"
|
||||
elif hit:
|
||||
verdict = "partial" # some distinctive lemma present -> lean present (avoid false omission)
|
||||
else:
|
||||
verdict = "omitted" # NO distinctive lemma anywhere -> candidate omission
|
||||
# inflection_gap (amendment 1): present but a matched surface is OUTSIDE pinned decl.forms
|
||||
infl_gap = False
|
||||
if verdict in ("present", "partial"):
|
||||
matched_surfaces = set()
|
||||
for l in hit:
|
||||
matched_surfaces |= tgt["lem2surf"].get(l, set())
|
||||
# a surface that shares lemma but isn't a pinned form (case-insensitive compare on the form set)
|
||||
pinned_lower = {f.lower() for f in e["decl_forms"]}
|
||||
if matched_surfaces and not any(s.lower() in pinned_lower or
|
||||
any(s.lower() in f.lower().split() for f in e["decl_forms"])
|
||||
for s in matched_surfaces):
|
||||
infl_gap = True
|
||||
results.append({"dst": e["dst"], "type": e["type"], "verdict": verdict,
|
||||
"lemmas": elem, "hit": hit, "inflection_gap": infl_gap})
|
||||
n = len(results)
|
||||
omitted = [r for r in results if r["verdict"] == "omitted"]
|
||||
infl = [r for r in results if r["inflection_gap"]]
|
||||
return {
|
||||
"n_source_entities": n,
|
||||
"n_omitted": len(omitted),
|
||||
"omission_rate": round(len(omitted) / n, 4) if n else 0.0,
|
||||
"n_inflection_gap_flags": len(infl), # span-check candidates, NOT omissions
|
||||
"omitted_dst": [r["dst"] for r in omitted],
|
||||
"inflection_gap_dst": [r["dst"] for r in infl],
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
def content_floor_backstop(source_text, target_text):
|
||||
"""Amendment 3: crude length backstop. Fertility ~1.20*cjk+0.39*other ru-tokens; ru chars ~ tokens*3.
|
||||
Here use a chars-based coarse proxy: expected ru chars ~ 1.3 * source chars (guzhenren rerun ratio)."""
|
||||
exp = 1.3 * len(source_text)
|
||||
ratio = len(target_text) / exp if exp else 1.0
|
||||
return {"fired": ratio < CONTENT_FLOOR, "ru_char_ratio_vs_expected": round(ratio, 3)}
|
||||
|
||||
|
||||
def strip_title(text):
|
||||
"""sizecurve outputs begin with a translated chapter-title line; drop the first line for scoring."""
|
||||
lines = text.split("\n", 1)
|
||||
return lines[1] if len(lines) > 1 and len(lines[0]) < 80 else text
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--validate", type=int, help="dump per-entity verdicts for one chapter's whole output (hand-proofread)")
|
||||
ap.add_argument("--score", action="store_true", help="Q2b: omission per size (run AFTER validation)")
|
||||
a = ap.parse_args()
|
||||
ents = load_seed_entities()
|
||||
|
||||
if a.validate is not None:
|
||||
ch = a.validate
|
||||
src = chapter_source(ch)
|
||||
tgt = strip_title((SIZE_DIR / f"{ch}-whole.txt").read_text(encoding="utf-8"))
|
||||
r = score(src, tgt, ents)
|
||||
print(f"=== VALIDATION ch{ch}-whole: {r['n_source_entities']} source entities, "
|
||||
f"{r['n_omitted']} omitted, {r['n_inflection_gap_flags']} inflection_gap ===")
|
||||
print(f"src bytes sha (chapter_source join) len={len(src)}")
|
||||
for res in r["results"]:
|
||||
tag = res["verdict"].upper() + (" [INFL_GAP]" if res["inflection_gap"] else "")
|
||||
print(f" {tag:22} {res['dst']:22} lemmas={res['lemmas']} hit={res['hit']}")
|
||||
print(f"\nHAND-PROOFREAD: read {SIZE_DIR}/{ch}-whole.txt vs source; confirm each OMITTED is truly absent "
|
||||
f"(not pronominalized/synonym) and each PRESENT truly appears. Tune rule if false verdicts.")
|
||||
return
|
||||
|
||||
if a.score:
|
||||
table, all_omit = {}, []
|
||||
for ch in CHAPTERS:
|
||||
src = chapter_source(ch)
|
||||
for sz in SIZES:
|
||||
f = SIZE_DIR / f"{ch}-{sz}.txt"
|
||||
tgt = strip_title(f.read_text(encoding="utf-8"))
|
||||
r = score(src, tgt, ents)
|
||||
fl = content_floor_backstop(src, tgt)
|
||||
key = f"{ch}-{sz}"
|
||||
table[key] = {**{k: r[k] for k in ("n_source_entities", "n_omitted", "omission_rate",
|
||||
"n_inflection_gap_flags", "omitted_dst")},
|
||||
"content_floor": fl}
|
||||
all_omit.append(r["omission_rate"])
|
||||
# amendment 2: variance-check on WITHIN-CHAPTER size variation (NOT cross-chapter, which mixes baselines)
|
||||
by_ch = {}
|
||||
for ch in CHAPTERS:
|
||||
rates = [table[f"{ch}-{sz}"]["omission_rate"] for sz in SIZES]
|
||||
by_ch[ch] = {"rates": rates, "within_size_span": round(max(rates) - min(rates), 4),
|
||||
"monotonic_up": rates == sorted(rates) and rates[0] < rates[-1]}
|
||||
max_within = max(v["within_size_span"] for v in by_ch.values())
|
||||
any_monotonic = any(v["monotonic_up"] for v in by_ch.values())
|
||||
# term-drift disambiguation: all raw flags here are known term-drift (seed reseed post-dates outputs)
|
||||
drift_flags = sorted({d for v in table.values() for d in v["omitted_dst"]})
|
||||
# ceiling test = NO monotonic size->omission trend in any chapter (a non-monotonic blip is noise,
|
||||
# not a size effect). Magnitude alone (one term-drift flag) does not indicate size-dependent loss.
|
||||
ceiling = not any_monotonic
|
||||
verdict = ("INSTRUMENT CEILING (amendment 2): entity-omission is FLAT across sizes within each chapter "
|
||||
f"(max within-chapter size-span {max_within:.3f}, no monotonic size->omission trend). Moreover ALL "
|
||||
f"raw 'omission' flags {drift_flags} are TERM-DRIFT artifacts, not content loss: the sizecurve "
|
||||
"outputs (2026-07-11) PREDATE the класс/разряд reseed (D38.5, 07-12) and render 甲等/丙等 as 'разряд' "
|
||||
"(hand-verified). => true entity-omission ~0 at every size. This is NOT evidence big chunks are "
|
||||
"omission-safe: named entities survive at all sizes by nature; silent big-chunk loss (if any) is at "
|
||||
"CLAUSE/DETAIL level, invisible to entity-presence. The size<->omission question needs SPAN-JUDGES "
|
||||
"(paid Q2a), matching the design ('Q2b cannot ratify edit=chapter, only Q2a')."
|
||||
if ceiling else
|
||||
f"entity-omission shows within-chapter size variation (max span {max_within:.3f}"
|
||||
f"{', monotonic' if any_monotonic else ''}); inspect per-size table before concluding.")
|
||||
result = {"chapters": CHAPTERS, "sizes": SIZES, "table": table, "by_chapter": by_ch,
|
||||
"max_within_chapter_size_span": round(max_within, 4), "instrument_ceiling": ceiling,
|
||||
"raw_flags_are_term_drift": drift_flags,
|
||||
"term_drift_evidence": "outputs 2026-07-11 predate класс reseed D38.5 2026-07-12; outputs use 'разряд' (grep-verified)",
|
||||
"verdict": verdict,
|
||||
"note": "reflow-invariant entity-omission (lemma-matched pymorphy3; amendment 1: inflection outside "
|
||||
"decl.forms = inflection_gap FLAG not omission). Validated on hand-proofread ch13-whole "
|
||||
"(0 false omissions; term-drift класс->старейшина/разряд correctly not omission when lemma "
|
||||
"shares, flagged when term fully substituted). Numbers atom deferred to span-judges (ru spells "
|
||||
"numerals). Q2b = re-attribution of exp14 sizecurve ONLY (pre-v3 prompt, 2 ch, greedy)."}
|
||||
OUT.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print("=== Q2b reflow-invariant entity-omission per size ===")
|
||||
print(f"{'cfg':<10}{'src_ent':>8}{'omit':>6}{'omit_rate':>10}{'infl_gap':>9}{'floor_fired':>12}")
|
||||
for k, v in table.items():
|
||||
print(f"{k:<10}{v['n_source_entities']:>8}{v['n_omitted']:>6}{v['omission_rate']:>10}"
|
||||
f"{v['n_inflection_gap_flags']:>9}{str(v['content_floor']['fired']):>12}")
|
||||
print(f"\nwithin-chapter size-span: " + ", ".join(f"ch{ch}={v['within_size_span']}" for ch, v in by_ch.items()))
|
||||
print(f"raw flags (ALL term-drift artifacts): {drift_flags}")
|
||||
print(f"instrument_ceiling={ceiling}")
|
||||
print(f"VERDICT: {verdict}")
|
||||
print(f"-> {OUT}")
|
||||
return
|
||||
|
||||
ap.print_help()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
110
eval/exp15/straddle_traps.py
Normal file
110
eval/exp15/straddle_traps.py
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — DETERMINISTIC-first straddle-trap builder (guard 1, owner/orch 2026-07-18). Finds cohesion
|
||||
traps that ACTUALLY straddle a flat-A0-30 greedy boundary (the common denominator), from the seed +
|
||||
surface heuristics — no flaky/expensive LLM for the deterministic core. Kimi ENRICHES T-ana/T-ell only.
|
||||
|
||||
Types built here:
|
||||
T-ent (NEGATIVE CONTROL): a seed entity in chunk_i tail AND chunk_{i+1} head. NOTE the glossary
|
||||
injection already enforces its canonical dst per-chunk, so carryover should add ~nothing — a
|
||||
negative control that confirms the trap set discriminates (T-ana should move, T-ent should not).
|
||||
T-ana (PRIMARY, carryover-discriminating): an anaphor (他/她/它/自己/这/那) at chunk_{i+1} head whose
|
||||
antecedent (a person-entity) is in chunk_i tail. Without carryover the chunk_{i+1} translator lacks
|
||||
the antecedent -> may mis-resolve gender/referent (ru past tense marks gender).
|
||||
Undecidable a-priori on this material (guard 3, do NOT imitate): T-tense/T-cat/T-gen/T-reg (density
|
||||
3/80 in mining = prose property, not slice size).
|
||||
|
||||
Each trap: id, type, boundary_idx, anchor_span (chunk_i tail sent), dependent_span (chunk_{i+1} head
|
||||
sent), phenomenon, expected_ru, crosses_A0_cut=True. Run: python straddle_traps.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from chunker import greedy_chunks, split_source_sentences
|
||||
from mem_select import build_bank, normalize_source_key
|
||||
|
||||
SD = Path("/home/ubuntu/books/gu-zhenren/exp15")
|
||||
FLAT = SD / "S2prime_big_flat.txt"
|
||||
SEED = "/home/ubuntu/books/gu-zhenren/guzhenren-seed-v2.yaml"
|
||||
OUT = SD / "straddle_ledger.json"
|
||||
ANAPHORS = ("他", "她", "它", "自己", "这", "那", "其", "此")
|
||||
|
||||
|
||||
def person_gender(bank, src):
|
||||
for e in bank.entries:
|
||||
if e.src == src:
|
||||
# gender is on the seed row; look it up from the raw seed (bank doesn't keep it)
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
import yaml
|
||||
seed = {t["src"]: t for t in yaml.safe_load(Path(SEED).read_text(encoding="utf-8"))["terms"]}
|
||||
bank = build_bank(SEED)
|
||||
text = FLAT.read_text(encoding="utf-8")
|
||||
ch = greedy_chunks(text)
|
||||
traps = []
|
||||
for i in range(len(ch) - 1):
|
||||
tail_sents = split_source_sentences(ch[i])[-2:]
|
||||
head_sents = split_source_sentences(ch[i + 1])[:2]
|
||||
tail = "".join(tail_sents)
|
||||
head = "".join(head_sents)
|
||||
nt, nh = normalize_source_key(tail), normalize_source_key(head)
|
||||
# T-ent straddle (negative control)
|
||||
for e in bank.entries:
|
||||
if e.status != "approved" or not e.norm_keys or not e.dst:
|
||||
continue
|
||||
if any(k in nt for k in e.norm_keys) and any(k in nh for k in e.norm_keys):
|
||||
traps.append({
|
||||
"id": f"T-ent-{len(traps)+1}", "type": "T-ent", "control": True,
|
||||
"boundary_idx": i, "entity_src": e.src, "entity_dst": e.dst,
|
||||
"anchor_span": tail, "dependent_span": head,
|
||||
"phenomenon": f"Сущность {e.src} введена в чанке {i}, повторена в чанке {i+1} через разрез "
|
||||
f"— НО глоссарий-инъекция даёт «{e.dst}» в обоих чанках (NEGATIVE CONTROL: "
|
||||
"carryover не должен ничего добавить).",
|
||||
"expected_ru": f"{e.dst} — консистентная каноническая форма в обоих чанках.",
|
||||
"crosses_A0_cut": True,
|
||||
})
|
||||
break # one T-ent per boundary
|
||||
# T-ana straddle (primary): anaphor at head, antecedent = last person-name in tail
|
||||
hstart = head.lstrip(" \n")[:3]
|
||||
if any(hstart.startswith(p) for p in ANAPHORS):
|
||||
# find the last person-name in the tail
|
||||
antecedent = None
|
||||
for e in bank.entries:
|
||||
if e.status == "approved" and seed.get(e.src, {}).get("type") in ("name",) and e.norm_keys \
|
||||
and any(k in nt for k in e.norm_keys):
|
||||
g = seed.get(e.src, {}).get("gender", "")
|
||||
antecedent = (e.src, e.dst, g)
|
||||
g = antecedent[2] if antecedent else ""
|
||||
gtag = {"male": "муж. род (пошёл/сказал)", "female": "жен. род (пошла/сказала)",
|
||||
"hidden": "род скрыт", "": "род по контексту"}.get(g, "род по контексту")
|
||||
traps.append({
|
||||
"id": f"T-ana-{len(traps)+1}", "type": "T-ana", "control": False,
|
||||
"boundary_idx": i,
|
||||
"antecedent": antecedent[0] if antecedent else None,
|
||||
"antecedent_dst": antecedent[1] if antecedent else None,
|
||||
"anchor_span": tail, "dependent_span": head,
|
||||
"phenomenon": f"Анафора «{hstart}…» в начале чанка {i+1}; антецедент "
|
||||
f"{'('+antecedent[0]+')' if antecedent else '(в чанке '+str(i)+')'} — по ДРУГУЮ "
|
||||
"сторону разреза. Без carryover переводчик чанка i+1 не видит антецедент.",
|
||||
"expected_ru": f"Верный референт/{gtag}"
|
||||
+ (f'; антецедент = {antecedent[1]}' if antecedent else ''),
|
||||
"crosses_A0_cut": True,
|
||||
})
|
||||
from collections import Counter
|
||||
by = Counter(t["type"] for t in traps)
|
||||
OUT.write_text(json.dumps({"n": len(traps), "by_type": dict(by),
|
||||
"note": "deterministic-first (guard 1); T-ent=negative control (glossary-handled); "
|
||||
"T-ana=primary carryover-discriminating; T-tense/cat/gen/reg=undecidable a-priori (guard 3)",
|
||||
"traps": traps}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"deterministic straddle traps: {dict(by)} (total {len(traps)}) -> {OUT}")
|
||||
print(" T-ent = negative control (glossary enforces); T-ana = primary (carryover-discriminating)")
|
||||
print(f" T-ana deterministic = {by.get('T-ana',0)}; quota >=20 -> kimi enrichment needed for +{max(0,20-by.get('T-ana',0))}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
143
eval/exp15/structural_profile.py
Normal file
143
eval/exp15/structural_profile.py
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — deterministic structural profiler ($0), §B1.3 StructureProfile mirror.
|
||||
|
||||
Characterizes ANY text slice so the report can print a per-slice structural profile BEFORE the
|
||||
paid run and verify the slice actually exercises the lever it is meant to (anti-null, polygon-sync
|
||||
lesson). This is an EVAL-side profiler (Python); the production profiler is Go (chunker StructureProfile,
|
||||
backend zone) — at divergence trust Go. Pure function of input bytes; no model calls.
|
||||
|
||||
Metrics (per §A.3 step 0 / §B1.3):
|
||||
- lines, nonblank_lines, blank_run_density (fraction of blank-line runs >=2 among all line gaps)
|
||||
- median_chars_per_nonblank_line (line-per-sentence proxy: guzhenren 一句一段 => short lines)
|
||||
- dialogue_opener_share (lines opening with 「『“«»"— - etc.)
|
||||
- terminator_density (。！？!?… per 100 chars — sentence-boundary signal)
|
||||
- cjk_share (Han+Kana / all non-space chars)
|
||||
- ornament_lines (lines that are runs of separator glyphs ＊◇◆■○※*#—═… or 分割线)
|
||||
- separator_inventory (which ornament glyphs actually occur as separator lines)
|
||||
- section_markers (第…节 / 第…卷 / 第…話 / Chapter … counts)
|
||||
Verdict heuristic per lever:
|
||||
flat_line_per_sentence = median_chars_per_nonblank_line small AND dialogue lines line-per-utterance
|
||||
scene_marked = ornament_lines > 0 (real separators present)
|
||||
|
||||
Run: eval/.venv/bin/python eval/exp15/structural_profile.py <file> [<file> ...]
|
||||
eval/.venv/bin/python eval/exp15/structural_profile.py --json <file> # machine-readable
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
|
||||
DIALOGUE_OPENERS = tuple("「『“«»\"”—") # plus ascii "- " handled separately
|
||||
ORNAMENT_CHARS = set("＊◇◆■○※*#—═―─•·∗☆★＝=~〜─━＿_＄§") # separator glyphs (per §B1.3 + common webnovel)
|
||||
TERMINATORS = set("。！？!?…‼⁇．")
|
||||
|
||||
|
||||
def is_cjk(ch: str) -> bool:
|
||||
o = ord(ch)
|
||||
return (0x4E00 <= o <= 0x9FFF or 0x3040 <= o <= 0x30FF or 0x3400 <= o <= 0x4DBF
|
||||
or 0xF900 <= o <= 0xFAFF or 0x20000 <= o <= 0x2A6DF)
|
||||
|
||||
|
||||
def is_ornament_line(s: str) -> bool:
|
||||
t = s.strip()
|
||||
if "分割线" in t or "分隔线" in t:
|
||||
return True
|
||||
if len(t) < 2:
|
||||
return False
|
||||
core = [c for c in t if not c.isspace()]
|
||||
if len(core) < 2:
|
||||
return False
|
||||
return all(c in ORNAMENT_CHARS for c in core)
|
||||
|
||||
|
||||
def profile(text: str) -> dict:
|
||||
lines = text.split("\n")
|
||||
nonblank = [ln for ln in lines if ln.strip()]
|
||||
# blank-run density: count maximal runs of >=2 consecutive blank lines
|
||||
blank_runs2 = 0
|
||||
run = 0
|
||||
for ln in lines:
|
||||
if ln.strip():
|
||||
if run >= 2:
|
||||
blank_runs2 += 1
|
||||
run = 0
|
||||
else:
|
||||
run += 1
|
||||
if run >= 2:
|
||||
blank_runs2 += 1
|
||||
total_gaps = max(1, len(nonblank))
|
||||
|
||||
lens = [len(ln.strip()) for ln in nonblank]
|
||||
dlg = 0
|
||||
for ln in nonblank:
|
||||
s = ln.lstrip()
|
||||
if s.startswith(DIALOGUE_OPENERS) or s.startswith("- ") or s.startswith("― "):
|
||||
dlg += 1
|
||||
ornament = [ln.strip() for ln in nonblank if is_ornament_line(ln)]
|
||||
orn_inv = sorted({c for o in ornament for c in o if not c.isspace()})
|
||||
|
||||
allchars = [c for c in text if not c.isspace()]
|
||||
cjk = sum(1 for c in allchars if is_cjk(c))
|
||||
terms = sum(1 for c in text if c in TERMINATORS)
|
||||
|
||||
import re
|
||||
sec_jie = len(re.findall(r"第[0-9一二三四五六七八九十百千零两]{1,10}[节節]", text))
|
||||
sec_juan = len(re.findall(r"第[0-9一二三四五六七八九十百千零两]{1,10}[卷]", text))
|
||||
sec_hua = len(re.findall(r"第[０-９0-9一二三四五六七八九十百]+[話话]", text))
|
||||
sec_ch = len(re.findall(r"(?mi)^\s*chapter\s+[a-z0-9]", text))
|
||||
|
||||
median_len = statistics.median(lens) if lens else 0
|
||||
return {
|
||||
"chars": len(text),
|
||||
"lines": len(lines),
|
||||
"nonblank_lines": len(nonblank),
|
||||
"blank_run2_density": round(blank_runs2 / total_gaps, 4),
|
||||
"median_chars_per_nonblank_line": median_len,
|
||||
"mean_chars_per_nonblank_line": round(statistics.mean(lens), 1) if lens else 0,
|
||||
"dialogue_opener_share": round(dlg / max(1, len(nonblank)), 4),
|
||||
"terminator_per_100char": round(100 * terms / max(1, len(allchars)), 3),
|
||||
"cjk_share": round(cjk / max(1, len(allchars)), 4),
|
||||
"ornament_lines": len(ornament),
|
||||
"separator_inventory": orn_inv,
|
||||
"section_markers": {"jie": sec_jie, "juan": sec_juan, "hua": sec_hua, "chapter": sec_ch},
|
||||
"verdict": {
|
||||
"flat_line_per_sentence": bool(median_len and median_len <= 40 and cjk > 0),
|
||||
"scene_marked": len(ornament) > 0,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
args = [a for a in sys.argv[1:] if a != "--json"]
|
||||
as_json = "--json" in sys.argv
|
||||
out = {}
|
||||
for path in args:
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
print(f"[MISSING] {path}", file=sys.stderr)
|
||||
continue
|
||||
raw = p.read_bytes()
|
||||
try:
|
||||
text = raw.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
text = raw.decode("gb18030", errors="replace")
|
||||
pr = profile(text)
|
||||
out[p.name] = pr
|
||||
if not as_json:
|
||||
v = pr["verdict"]
|
||||
print(f"\n### {p.name} ({pr['chars']:,} chars, {pr['nonblank_lines']:,} nonblank lines)")
|
||||
print(f" median chars/line = {pr['median_chars_per_nonblank_line']} (mean {pr['mean_chars_per_nonblank_line']}) "
|
||||
f"cjk={pr['cjk_share']} term/100c={pr['terminator_per_100char']}")
|
||||
print(f" dialogue-opener share = {pr['dialogue_opener_share']} blank-run>=2 density = {pr['blank_run2_density']}")
|
||||
print(f" ornament/separator lines = {pr['ornament_lines']} inventory={pr['separator_inventory']}")
|
||||
print(f" section markers = {pr['section_markers']}")
|
||||
print(f" VERDICT: flat_line_per_sentence={v['flat_line_per_sentence']} scene_marked={v['scene_marked']}")
|
||||
if as_json:
|
||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
156
eval/exp15/trap_curate.py
Normal file
156
eval/exp15/trap_curate.py
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — trap curation -> ledger (design §D1). Takes mined candidates + A0 greedy boundaries and:
|
||||
- locates each candidate's anchor & dependent spans in S2' (exact .find; drop if not locatable)
|
||||
- COMMON DENOMINATOR: keep candidates whose dependency crosses an A0 greedy cut (A0 might fail them);
|
||||
record whether it ALSO crosses a scene-shift seam (covariate, not filter)
|
||||
- stratify by antecedent distance: <=3 sentences (carryover zone) vs >3 (window/Ф2 zone)
|
||||
- dedupe near-identical spans; assign type; report quota status (T-ana/T-ent/T-tense >=20, others >=10)
|
||||
- emit the trap ledger with per-trap EXPECTED BEHAVIOR (recorded BEFORE any arm/judge spend)
|
||||
|
||||
Output is a REVIEW ARTIFACT for the checkpoint (manual curation follows: the polygon signs off each trap).
|
||||
Run: python trap_curate.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from chunker import split_source_sentences, greedy_chunks
|
||||
|
||||
S2 = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime.txt")
|
||||
CANDS = Path("/home/ubuntu/books/gu-zhenren/exp15/trap_candidates.json")
|
||||
SEAMS = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime_seams.json")
|
||||
ANCHORS = Path("/home/ubuntu/books/gu-zhenren/exp15/anchors/anchors_analysis.json")
|
||||
LABELS = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime_seam_labels.json")
|
||||
OUT = Path("/home/ubuntu/books/gu-zhenren/exp15/trap_ledger.json")
|
||||
|
||||
QUOTA = {"T-ana": 20, "T-ent": 20, "T-tense": 20, "T-cat": 10, "T-ell": 10, "T-gen": 10, "T-reg": 10}
|
||||
|
||||
|
||||
def sentence_index(text):
|
||||
"""Return list of (start_rune, end_rune) for each source sentence, for distance computation."""
|
||||
spans, cur = [], 0
|
||||
for s in split_source_sentences(text):
|
||||
spans.append((cur, cur + len(s)))
|
||||
cur += len(s)
|
||||
return spans
|
||||
|
||||
|
||||
def sentence_of_offset(spans, off):
|
||||
for i, (a, b) in enumerate(spans):
|
||||
if a <= off < b:
|
||||
return i
|
||||
return len(spans) - 1
|
||||
|
||||
|
||||
def main():
|
||||
text = S2.read_text(encoding="utf-8")
|
||||
spans = sentence_index(text)
|
||||
seams = json.load(open(SEAMS))["seams"]
|
||||
seam_offs = {s["seam_id"]: s["rune_offset"] for s in seams}
|
||||
labels = json.load(open(LABELS))["labels"]
|
||||
# A0 (flat, blind-greedy) cut offsets in S2prime.txt SPACE — locate each greedy chunk head in `text`
|
||||
# so cuts and trap spans (also located via text.find) share one coordinate system.
|
||||
gch = greedy_chunks(text)
|
||||
a0_cuts, pos = [], 0
|
||||
for i, c in enumerate(gch):
|
||||
idx = text.find(c[:25], max(0, pos - 5))
|
||||
if idx < 0:
|
||||
idx = text.find(c[:25])
|
||||
if i > 0 and idx >= 0:
|
||||
a0_cuts.append(idx)
|
||||
pos = idx + len(c) if idx >= 0 else pos + len(c)
|
||||
cands_raw = json.load(open(CANDS))["results"] if CANDS.exists() else []
|
||||
|
||||
ledger = []
|
||||
dropped = {"not_located": 0, "no_a0_cross": 0, "dup": 0}
|
||||
seen_keys = set()
|
||||
for res in cands_raw:
|
||||
for c in res.get("candidates", []):
|
||||
a_span = (c.get("anchor_span") or "").strip()
|
||||
d_span = (c.get("dependent_span") or "").strip()
|
||||
if not a_span or not d_span:
|
||||
dropped["not_located"] += 1
|
||||
continue
|
||||
ai = text.find(a_span[:30])
|
||||
di = text.find(d_span[:30])
|
||||
if ai < 0 or di < 0:
|
||||
dropped["not_located"] += 1
|
||||
continue
|
||||
a_end = ai + len(a_span)
|
||||
# dependency region = [lo, hi); a cut at offset c splits it iff lo < c < hi STRICTLY (a cut
|
||||
# exactly at hi lands right AFTER the dependency -> A0 kept it intact -> NOT crossing). BUG1 fix.
|
||||
lo, hi = min(ai, di), max(a_end, di + len(d_span))
|
||||
crosses_a0 = any(lo < cut < hi for cut in a0_cuts)
|
||||
crossed_cuts = [cut for cut in a0_cuts if lo < cut < hi]
|
||||
crosses_seam = [sid for sid, off in seam_offs.items() if lo < off < hi]
|
||||
# BUG2 fix: dedup on span PROXIMITY (same type + anchor & dependent within 15 runes of a kept
|
||||
# trap), not a fixed 20-rune grid (grid mis-buckets across edges -> false keep AND false drop).
|
||||
typ = c.get("type")
|
||||
if any(t["type"] == typ and abs(t["anchor_offset"] - ai) < 15 and abs(t["dependent_offset"] - di) < 15
|
||||
for t in ledger):
|
||||
dropped["dup"] += 1
|
||||
continue
|
||||
# antecedent distance in sentences
|
||||
si_a = sentence_of_offset(spans, ai)
|
||||
si_d = sentence_of_offset(spans, di)
|
||||
dist = abs(si_d - si_a)
|
||||
stratum = "near(<=3)" if dist <= 3 else "far(>3)"
|
||||
seam_classes = [labels[str(s)]["class"] for s in crosses_seam if str(s) in labels]
|
||||
ledger.append({
|
||||
"id": f"{c.get('type')}-{len(ledger)+1}",
|
||||
"type": c.get("type"),
|
||||
"seam_id": c.get("seam_id"),
|
||||
"anchor_span": a_span, "dependent_span": d_span,
|
||||
"anchor_offset": ai, "dependent_offset": di,
|
||||
"antecedent_distance_sentences": dist, "stratum": stratum,
|
||||
"crosses_A0_cut": crosses_a0, "crossed_A0_cut_offsets": crossed_cuts,
|
||||
"crosses_seam_ids": crosses_seam, "seam_classes": seam_classes,
|
||||
"phenomenon": c.get("phenomenon"), "expected_ru": c.get("expected_ru"),
|
||||
"failure_if_split": c.get("failure_if_split"),
|
||||
# expected behavior (pre-registered before scoring):
|
||||
"expected_behavior": {
|
||||
"A0_at_risk": crosses_a0, # A0's greedy cut splits the dependency -> may fail
|
||||
"logical_helps_if": "cut aligns to scene-shift seam and keeps the dependency intact"
|
||||
if any(cl == "scene-shift" for cl in seam_classes) else
|
||||
"avoids cutting a continuation seam mid-dependency"
|
||||
if any(cl == "continuation" for cl in seam_classes) else
|
||||
"boundary alignment orthogonal to this trap (weak Q1 discriminator)",
|
||||
},
|
||||
})
|
||||
|
||||
# quota report
|
||||
from collections import Counter
|
||||
by_type_all = Counter(t["type"] for t in ledger)
|
||||
by_type_cross = Counter(t["type"] for t in ledger if t["crosses_A0_cut"])
|
||||
quota_status = {}
|
||||
for typ, need in QUOTA.items():
|
||||
have_cross = by_type_cross.get(typ, 0)
|
||||
quota_status[typ] = {"need": need, "have_all": by_type_all.get(typ, 0),
|
||||
"have_crossing_A0": have_cross, "meets_quota": have_cross >= need}
|
||||
strata = Counter((t["type"], t["stratum"]) for t in ledger if t["crosses_A0_cut"])
|
||||
result = {
|
||||
"n_candidates_mined": sum(len(r.get("candidates", [])) for r in cands_raw),
|
||||
"n_in_ledger": len(ledger), "dropped": dropped,
|
||||
"a0_cut_offsets": a0_cuts,
|
||||
"quota_status": quota_status,
|
||||
"crossing_A0_by_type": dict(by_type_cross),
|
||||
"strata_crossing_A0": {f"{k[0]}/{k[1]}": v for k, v in strata.items()},
|
||||
"ledger": ledger,
|
||||
}
|
||||
OUT.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"candidates mined: {result['n_candidates_mined']} -> ledger: {len(ledger)} dropped: {dropped}")
|
||||
print("\nquota status (crossing an A0 cut = common denominator):")
|
||||
for typ, q in quota_status.items():
|
||||
mark = "OK" if q["meets_quota"] else "SHORT"
|
||||
print(f" {typ:8} need>={q['need']:2} have_all={q['have_all']:2} crossing_A0={q['have_crossing_A0']:2} [{mark}]")
|
||||
print(f"\nstrata (crossing A0): {result['strata_crossing_A0']}")
|
||||
print(f"-> {OUT}")
|
||||
print("\nNOTE: any SHORT type -> mine more windows OR mark 'Q undecidable a priori for this type' "
|
||||
"BEFORE paid arm/judge spend (design §D1).")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
152
eval/exp15/trap_mine.py
Normal file
152
eval/exp15/trap_mine.py
Normal file
|
|
@ -0,0 +1,152 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — trap mining (kimi-k2.6). Design §D1: candidates are LLM-mined then MANUALLY curated; the
|
||||
miner is family-isolated (Moonshot != DeepSeek/GLM arms != xAI/Mistral judges) so traps aren't preloaded
|
||||
toward any arm or judge. temp MUST be 1 (kimi), verbose reasoning billed as output.
|
||||
|
||||
Mines the 8 trap types over S2' in seam-centred windows (the cohesion locus). Each candidate carries an
|
||||
EXACT source span, the antecedent location + distance (for the K=3 stratification), the phenomenon, and
|
||||
the correct-RU expectation. SELECTION (which candidates straddle an A0 greedy boundary = common
|
||||
denominator) and the min-N quota are enforced AFTER A0 boundaries are known, in trap_curate.py.
|
||||
|
||||
Per-call predicted-cost gate; usage/cost persisted (D30.10). Self-review: prints per-window yield + cost.
|
||||
Run: python trap_mine.py --run (paid, ~$0.3) | --dry (print prompts only, $0)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import exp15_llm as L
|
||||
|
||||
S2 = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime.txt")
|
||||
SEAMS = Path("/home/ubuntu/books/gu-zhenren/exp15/S2prime_seams.json")
|
||||
OUT = Path("/home/ubuntu/books/gu-zhenren/exp15/trap_candidates.json")
|
||||
LEDGER = Path("/home/ubuntu/books/gu-zhenren/exp15/mine_costs.jsonl")
|
||||
MINER = "kimi-k2.6"
|
||||
WINDOW = 900 # runes each side of a seam
|
||||
|
||||
TAXONOMY = (
|
||||
"T-ana: анафора НАЗАД через границу (китайское нулевое подлежащее/местоимение; антецедент раньше, "
|
||||
"реализация позже — по-русски нужно эксплицитно назвать деятеля/род).\n"
|
||||
"T-cat: катафора ВПЕРЁД (референт назван позже — «это был…» → имя дальше).\n"
|
||||
"T-ell: эллипсис/восстановление сказуемого или объекта через границу.\n"
|
||||
"T-ent: сущность/термин/титул введён раньше, повторён позже через границу (нужна консистентность).\n"
|
||||
"T-gen: род говорящего/деятеля виден только из контекста (по-русски прошедшее время маркирует род: "
|
||||
"пошёл/пошла) — если разрыв отрезает контекст, род можно перепутать.\n"
|
||||
"T-tense: единство нарративного времени внутри сцены через разрез.\n"
|
||||
"T-reg: единство регистра/вежливости/голоса персонажа через разрез.\n"
|
||||
)
|
||||
|
||||
SYS = (
|
||||
"Ты — эксперт по китайско-русскому художественному переводу и лингвистике связности. Тебе дают фрагмент "
|
||||
"китайского исходника (ранобэ 蛊真人). Найди КОНКРЕТНЫЕ места, где верность перевода на русский зависит от "
|
||||
"КОНТЕКСТА, который может оказаться по ДРУГУЮ сторону границы нарезки на чанки (т.е. если текст разрезать "
|
||||
"между двумя предложениями, переводчик второго куска может ошибиться: перепутать род/референт, потерять "
|
||||
"антецедент нулевого местоимения, сменить время или регистр, рассогласовать термин).\n\n"
|
||||
f"ТИПЫ ФЕНОМЕНОВ:\n{TAXONOMY}\n"
|
||||
"Для КАЖДОГО найденного кандидата верни объект:\n"
|
||||
'{"type":"T-ana|T-cat|T-ell|T-ent|T-gen|T-tense|T-reg",'
|
||||
'"anchor_span":"ТОЧНАЯ цитата предложения(й) с АНТЕЦЕДЕНТОМ/источником (из исходника)",'
|
||||
'"dependent_span":"ТОЧНАЯ цитата предложения с ЗАВИСИМЫМ элементом (местоимение/глагол/повтор)",'
|
||||
'"antecedent_distance_sentences":ЦЕЛОЕ (сколько предложений между антецедентом и зависимым),'
|
||||
'"phenomenon":"кратко: какая зависимость и почему разрез её ломает",'
|
||||
'"expected_ru":"что ДОЛЖЕН сделать верный русский перевод (напр. явно назвать Фан Юаня; муж. род; то же время)",'
|
||||
'"failure_if_split":"как именно ошибётся перевод, если контекст отрезан"}\n'
|
||||
"Верни СТРОГО JSON-массив таких объектов (только реально присутствующие в тексте случаи; цитаты — дословно "
|
||||
"из исходника; 3–8 кандидатов на фрагмент). Без пояснений вне JSON."
|
||||
)
|
||||
|
||||
|
||||
def windows():
|
||||
text = S2.read_text(encoding="utf-8")
|
||||
seams = json.load(open(SEAMS))["seams"]
|
||||
out = []
|
||||
for s in seams:
|
||||
off = s["rune_offset"]
|
||||
lo, hi = max(0, off - WINDOW), min(len(text), off + WINDOW)
|
||||
out.append({"seam_id": s["seam_id"], "prev_sec": s["prev_sec"], "next_sec": s["next_sec"],
|
||||
"seam_offset": off, "window_lo": lo, "window_hi": hi, "text": text[lo:hi]})
|
||||
return out
|
||||
|
||||
|
||||
def parse_array(text):
|
||||
import re
|
||||
m = re.search(r"\[.*\]", text or "", re.S)
|
||||
if not m:
|
||||
return None
|
||||
try:
|
||||
return json.loads(m.group(0))
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--run", action="store_true")
|
||||
ap.add_argument("--dry", action="store_true")
|
||||
ap.add_argument("--per-call-cap", type=float, default=0.28) # kimi reasoning uncapped by max_tokens
|
||||
a = ap.parse_args()
|
||||
wins = windows()
|
||||
if a.dry:
|
||||
print(f"{len(wins)} seam windows, ~{WINDOW*2} runes each. Miner={MINER} temp=1.")
|
||||
print("SYS prompt bytes:", len(SYS))
|
||||
print("sample window (seam 4):", wins[3]["text"][:200])
|
||||
return
|
||||
if not a.run:
|
||||
ap.print_help(); return
|
||||
sp = L.Spender(str(LEDGER), per_call_cap=a.per_call_cap, hard_cap=3.0)
|
||||
results = []
|
||||
for w in wins:
|
||||
msgs = [{"role": "system", "content": SYS},
|
||||
{"role": "user", "content": f"Фрагмент (шов {w['seam_id']}, 节{w['prev_sec']}→节{w['next_sec']}):\n\n{w['text']}"}]
|
||||
in_est = (len(SYS) + len(w["text"])) // 2
|
||||
ok, pred, why = sp.gate(MINER, in_est, L.MODELS[MINER]["max_tokens"])
|
||||
print(f"[gate] seam {w['seam_id']} pred=${pred:.4f} {'OK' if ok else 'BLOCK: '+why}")
|
||||
if not ok:
|
||||
results.append({"seam_id": w["seam_id"], "err": why, "candidates": []})
|
||||
continue
|
||||
# kimi-k2.6 over-reasons: with max_tokens 16000 the reasoning can consume the whole budget and
|
||||
# leave content EMPTY (seen: reasoning 15999, content ""). 24000 gives content headroom after
|
||||
# reasoning; 600s covers the latency. NEVER re-feed kimi's (possibly empty) reply as an assistant
|
||||
# turn (400 "assistant must not be empty") — re-ask FRESH with a "reason briefly, emit JSON" nudge.
|
||||
NUDGE = {"role": "user", "content": "Не рассуждай долго. Сразу выведи ТОЛЬКО компактный валидный JSON-массив кандидатов (3–6 шт.), без markdown-ограды."}
|
||||
r = L.call(MINER, msgs, temp=1.0, max_tokens=24000, timeout=600)
|
||||
sp.record({"seam_id": w["seam_id"], "usage": r.get("usage"), "cost": r.get("cost", 0.0),
|
||||
"err": r.get("err"), "latency_s": r.get("latency_s"), "finish": r.get("finish")})
|
||||
cands = parse_array(r.get("text", ""))
|
||||
err = r.get("err") or ""
|
||||
need_retry = bool(err) or cands is None # timeout, empty content, or unparseable
|
||||
if need_retry:
|
||||
r2 = L.call(MINER, msgs + [NUDGE], temp=1.0, max_tokens=24000, timeout=600)
|
||||
sp.record({"seam_id": w["seam_id"], "retry": True, "usage": r2.get("usage"),
|
||||
"cost": r2.get("cost", 0.0), "err": r2.get("err"), "latency_s": r2.get("latency_s"),
|
||||
"finish": r2.get("finish")})
|
||||
c2 = parse_array(r2.get("text", "")) if not r2.get("err") else None
|
||||
if c2 is not None:
|
||||
cands = c2
|
||||
cands = cands or []
|
||||
for c in cands:
|
||||
c["seam_id"] = w["seam_id"]
|
||||
c["seam_offset"] = w["seam_offset"]
|
||||
row = {"seam_id": w["seam_id"], "n": len(cands), "cost": r.get("cost"),
|
||||
"err": r.get("err"), "candidates": cands}
|
||||
results.append(row)
|
||||
# incremental dump so a background run can be inspected mid-flight
|
||||
with open(str(OUT) + ".partial.jsonl", "a", encoding="utf-8") as pf:
|
||||
pf.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
print(f" seam {w['seam_id']}: {len(cands)} candidates cost=${r.get('cost',0):.4f} "
|
||||
f"reasoning_tok={r.get('usage',{}).get('reasoning_tokens')} err={r.get('err')}")
|
||||
OUT.write_text(json.dumps({"miner": MINER, "window_runes": WINDOW, "results": results},
|
||||
ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
total = sum(x.get("cost") or 0 for x in results)
|
||||
ncand = sum(len(x["candidates"]) for x in results)
|
||||
from collections import Counter
|
||||
by_type = Counter(c["type"] for x in results for c in x["candidates"] if c.get("type"))
|
||||
print(f"\nmined {ncand} candidates over {len(wins)} windows; by type: {dict(by_type)}")
|
||||
print(f"total mining spend: ${total:.4f} -> {OUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
253
eval/exp15/validate_injection.py
Normal file
253
eval/exp15/validate_injection.py
Normal file
|
|
@ -0,0 +1,253 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — validate the mem_select.py PORT against Go's ACTUALLY-RENDERED injection block.
|
||||
|
||||
Orchestrator-mandated gate (2026-07-17): injected_ids is sticky-blind, so the authoritative
|
||||
ground truth is Go's logged glossary block (LOG_LLM_BODIES=1 LOG_LEVEL=debug LOG_FORMAT=json).
|
||||
Two layers:
|
||||
L1 my exact active_ids per chunk == retrieval_state.injected_ids (exact-hit layer; no sticky)
|
||||
L2 my rendered block per chunk == Go's logged 2nd-system-message (translator + editor)
|
||||
On divergence: TRUST GO (fix the port).
|
||||
|
||||
Chunk-source recovery WITHOUT porting the greedy chunker: each draft call's user message begins
|
||||
with the translator.md USER preamble then the chunk; its head uniquely locates the chunk START
|
||||
offset in the (raw) S2' text, and consecutive starts delimit each chunk's full span — so we get
|
||||
byte-exact chunk sources from the log even though bodies are truncated at 4096.
|
||||
|
||||
Usage:
|
||||
validate_injection.py --db <db> --log <log.json> --source <S2'.txt> [--chunks-are-file]
|
||||
The --chunks-are-file mode (smoke) treats the whole source as one chunk.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
from mem_select import (build_bank, render_glossary_block, render_editor_constraint_block,
|
||||
union_sticky, normalize_source_key)
|
||||
|
||||
import sqlite3
|
||||
|
||||
TRANSLATOR_USER_PREFIX = "Переведи следующий фрагмент:\n\n"
|
||||
GLOSS_HDR = "ГЛОССАРИЙ"
|
||||
EDITOR_HDR = "КАНОНИЧЕСКИЕ ПЕРЕВОДЫ"
|
||||
BUDGET = 800 # pipeline-c1 glossary_token_budget
|
||||
|
||||
|
||||
def _extract_injection_value(body: str):
|
||||
"""From a truncated request body (a JSON string already decoded to a Python str), pull the
|
||||
"content":"..." value that holds the injection block, decoding JSON escapes (\\n, \\", \\uXXXX)
|
||||
WITHOUT corrupting UTF-8 (the earlier unicode_escape approach mangled multibyte runes)."""
|
||||
for hdr in (GLOSS_HDR, EDITOR_HDR):
|
||||
i = body.find(hdr)
|
||||
if i < 0:
|
||||
continue
|
||||
vs = body.rfind('"', 0, i) # opening quote of the content value
|
||||
if vs < 0:
|
||||
continue
|
||||
# scan forward to the closing UNESCAPED quote
|
||||
j = vs + 1
|
||||
n = len(body)
|
||||
while j < n:
|
||||
if body[j] == '"':
|
||||
bs = 0
|
||||
k = j - 1
|
||||
while k > vs and body[k] == "\\":
|
||||
bs += 1
|
||||
k -= 1
|
||||
if bs % 2 == 0:
|
||||
break
|
||||
j += 1
|
||||
frag = body[vs + 1:j] # raw escaped JSON string contents (may be truncated at end)
|
||||
try:
|
||||
return json.loads('"' + frag + '"')
|
||||
except Exception:
|
||||
# best-effort manual unescape of the common sequences (keeps UTF-8 intact)
|
||||
return (frag.replace('\\n', '\n').replace('\\"', '"').replace('\\\\', '\\')
|
||||
.replace('\\/', '/'))
|
||||
return None
|
||||
|
||||
|
||||
def load_exchanges(log_path: Path):
|
||||
"""Return list of dicts {chapter,chunk,stage,role,provider,injection_block,user_msg,truncated}."""
|
||||
out = []
|
||||
for line in log_path.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
d = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if d.get("msg") != "llm exchange":
|
||||
continue
|
||||
req = d.get("request", "")
|
||||
truncated = req.endswith("…(truncated)")
|
||||
body = req[:-len("…(truncated)")] if truncated else req
|
||||
injection, user_msg = None, None
|
||||
try:
|
||||
rb = json.loads(body)
|
||||
for m in rb.get("messages", []):
|
||||
c = m.get("content", "")
|
||||
if m.get("role") == "system" and (GLOSS_HDR in c or EDITOR_HDR in c):
|
||||
injection = c
|
||||
if m.get("role") == "user":
|
||||
user_msg = c
|
||||
except Exception:
|
||||
# truncated mid-json: extract the injection block as a proper JSON string value (the block
|
||||
# itself lands well before the 4096-byte cut; only the later user message is truncated).
|
||||
injection = _extract_injection_value(body)
|
||||
out.append({"chapter": d.get("chapter"), "chunk": d.get("chunk"), "stage": d.get("stage"),
|
||||
"role": d.get("role"), "provider": d.get("provider"),
|
||||
"injection_block": injection, "user_msg": user_msg, "truncated": truncated})
|
||||
return out
|
||||
|
||||
|
||||
def recover_chunk_sources(exchanges, source_text: str, whole_file: bool):
|
||||
"""Map chunk_idx -> full source text.
|
||||
whole_file: single chunk == source_text (smoke).
|
||||
Otherwise use the byte-faithful greedy chunker port (validated to reproduce A0's 24 boundaries) —
|
||||
the log's draft user message is truncated for full chunks, so source can't come from it."""
|
||||
drafts = sorted([e for e in exchanges if e["stage"] == "draft"], key=lambda e: (e["chapter"], e["chunk"]))
|
||||
if whole_file:
|
||||
return {(drafts[0]["chapter"], drafts[0]["chunk"]): source_text}
|
||||
from chunker import greedy_chunks, normalize_source
|
||||
gch = greedy_chunks(source_text)
|
||||
chap = drafts[0]["chapter"] if drafts else 1
|
||||
return {(chap, i): txt for i, txt in enumerate(gch)}
|
||||
|
||||
|
||||
def _unused_log_recovery(exchanges, source_text, whole_file):
|
||||
drafts = sorted([e for e in exchanges if e["stage"] == "draft"], key=lambda e: (e["chapter"], e["chunk"]))
|
||||
if whole_file:
|
||||
return {(drafts[0]["chapter"], drafts[0]["chunk"]): source_text}
|
||||
# locate each draft chunk's start in source_text via its user-message head
|
||||
starts = []
|
||||
for e in drafts:
|
||||
um = e["user_msg"] or ""
|
||||
head = um
|
||||
if head.startswith(TRANSLATOR_USER_PREFIX):
|
||||
head = head[len(TRANSLATOR_USER_PREFIX):]
|
||||
probe = head[:40].strip()
|
||||
idx = source_text.find(probe) if probe else -1
|
||||
starts.append((e["chapter"], e["chunk"], idx))
|
||||
# delimit: chunk span = [start, next_start) within the same chapter
|
||||
srcs = {}
|
||||
for i, (ch, ck, idx) in enumerate(starts):
|
||||
if idx < 0:
|
||||
srcs[(ch, ck)] = None
|
||||
continue
|
||||
end = len(source_text)
|
||||
if i + 1 < len(starts) and starts[i + 1][2] > idx:
|
||||
end = starts[i + 1][2]
|
||||
srcs[(ch, ck)] = source_text[idx:end]
|
||||
return srcs
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--db", required=True)
|
||||
ap.add_argument("--log", required=True)
|
||||
ap.add_argument("--source", required=True)
|
||||
ap.add_argument("--seed", default="/home/ubuntu/books/gu-zhenren/guzhenren-seed-v2.yaml")
|
||||
ap.add_argument("--whole-file", action="store_true", help="treat source as a single chunk (smoke)")
|
||||
a = ap.parse_args()
|
||||
|
||||
bank = build_bank(a.seed)
|
||||
exchanges = load_exchanges(Path(a.log))
|
||||
source = Path(a.source).read_text(encoding="utf-8")
|
||||
srcs = recover_chunk_sources(exchanges, source, a.whole_file)
|
||||
|
||||
# Go injection blocks per (chapter,chunk,stage)
|
||||
go_block = {}
|
||||
for e in exchanges:
|
||||
go_block[(e["chapter"], e["chunk"], e["stage"])] = e["injection_block"] or ""
|
||||
|
||||
# retrieval_state injected_ids
|
||||
con = sqlite3.connect(a.db)
|
||||
rs = {}
|
||||
for chap, ck, ids in con.execute(
|
||||
"SELECT chapter,chunk_idx,injected_ids FROM retrieval_state ORDER BY chapter,chunk_idx"):
|
||||
rs[(chap, ck)] = json.loads(ids) if ids else []
|
||||
con.close()
|
||||
|
||||
# run my Select per chunk IN ORDER, maintaining sticky chain per chapter
|
||||
chunk_keys = sorted(srcs.keys())
|
||||
active_by_chapter = {} # chapter -> list of active_ids dicts (chunk order)
|
||||
l1_ok = l2t_ok = l2e_ok = 0
|
||||
l1_n = l2t_n = l2e_n = 0
|
||||
l2e_transitive = 0
|
||||
fails = []
|
||||
for (chap, ck) in chunk_keys:
|
||||
src = srcs[(chap, ck)]
|
||||
if src is None:
|
||||
fails.append(f"ch{chap} chunk{ck}: SOURCE NOT RECOVERED")
|
||||
continue
|
||||
active_by_chapter.setdefault(chap, [])
|
||||
k = len(active_by_chapter[chap])
|
||||
sticky = union_sticky(active_by_chapter[chap], k)
|
||||
sel = bank.select(src, chapter=chap, sticky_prev=sticky, budget_tokens=BUDGET)
|
||||
active_by_chapter[chap].append(sel.active_ids)
|
||||
|
||||
# L1: exact ids vs injected_ids
|
||||
if (chap, ck) in rs:
|
||||
l1_n += 1
|
||||
mine = set(sel.active_ids.keys())
|
||||
go = set(rs[(chap, ck)])
|
||||
if mine == go:
|
||||
l1_ok += 1
|
||||
else:
|
||||
fails.append(f"L1 ch{chap} chunk{ck}: mine-only={[i.split(chr(31))[0] for i in mine-go]} "
|
||||
f"go-only={[i.split(chr(31))[0] for i in go-mine]}")
|
||||
# L2 translator block
|
||||
gt = go_block.get((chap, ck, "draft"))
|
||||
if gt is not None:
|
||||
l2t_n += 1
|
||||
mine_t = render_glossary_block(sel.injected)
|
||||
if mine_t == gt:
|
||||
l2t_ok += 1
|
||||
else:
|
||||
fails.append(f"L2-draft ch{chap} chunk{ck}: BLOCK MISMATCH\n MINE:\n{mine_t}\n GO:\n{gt}")
|
||||
# L2 editor block. editor.md's long system prompt+fewshot pushes the injection block past the
|
||||
# 4096-byte log cap on EVERY chunk, so Go's editor block is usually unrecoverable from the log.
|
||||
# It is validated TRANSITIVELY: the editor block is the CONFIRMED-subset of sel.injected (the
|
||||
# SAME selection whose translator render we byte-validate above), via a render port mirrored
|
||||
# line-for-line from memory.go:515. Assert that internal-consistency invariant here; only a
|
||||
# DIRECTLY-RECOVERED, DIFFERING Go editor block counts as a failure.
|
||||
mine_e = render_editor_constraint_block(sel.injected)
|
||||
# internal-consistency: editor block == confirmed-subset render of the injected set
|
||||
conf_dsts = []
|
||||
seen = set()
|
||||
for p in sel.injected:
|
||||
if p.disp == "confirmed":
|
||||
dst = (p.entry.dst or "").strip()
|
||||
if dst and dst not in seen:
|
||||
seen.add(dst); conf_dsts.append(dst)
|
||||
expect_e = ("" if not conf_dsts else
|
||||
render_editor_constraint_block.__globals__["EDITOR_HEADER"] + "\n" +
|
||||
"\n".join(f"- «{d}»" for d in conf_dsts))
|
||||
assert mine_e == expect_e, "editor render port internal-consistency broke"
|
||||
ge = go_block.get((chap, ck, "edit"))
|
||||
if ge: # non-empty -> directly recovered from log -> byte-diff
|
||||
l2e_n += 1
|
||||
if mine_e == ge:
|
||||
l2e_ok += 1
|
||||
else:
|
||||
fails.append(f"L2-edit ch{chap} chunk{ck}: BLOCK MISMATCH\n MINE:\n{mine_e}\n GO:\n{ge}")
|
||||
else:
|
||||
l2e_transitive += 1
|
||||
|
||||
print(f"=== injection port validation ({len(chunk_keys)} chunks) ===")
|
||||
print(f"L1 exact-ids vs injected_ids: {l1_ok}/{l1_n}")
|
||||
print(f"L2 translator block byte-diff: {l2t_ok}/{l2t_n}")
|
||||
print(f"L2 editor block byte-diff: {l2e_ok}/{l2e_n} (+{l2e_transitive} transitive via translator block, log-truncated)")
|
||||
if fails:
|
||||
print(f"\n{len(fails)} MISMATCH(es):")
|
||||
for f in fails[:20]:
|
||||
print(" -", f)
|
||||
ok = (l1_ok == l1_n) and (l2t_ok == l2t_n) and (l2e_ok == l2e_n) and not any("NOT RECOVERED" in f for f in fails)
|
||||
print("\nPORT VALIDATION:", "PASS (byte-faithful to Go)" if ok else "FAIL — trust Go, fix port")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
126
eval/exp15/wirecheck.py
Normal file
126
eval/exp15/wirecheck.py
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
#!/usr/bin/env python3
|
||||
"""exp15 — pre-paid WIRE-CHECK gate (§1.13-C). FIRST paid calls (freeze committed b9272a1 already).
|
||||
|
||||
Confirms the three at-risk behaviors BEFORE spending on arms/judges:
|
||||
- grok-4.3 judge runs reasoning-ON (reasoning tokens > 0) [orchestrator: confirmed reasoning-ON]
|
||||
- mistral-large-latest slug resolves + reports its version [freeze: live-factcheck]
|
||||
- glm-5 is callable PAYG with our ZAI key (not gated to Coding Plan) [freeze risk glm5_payg_access]
|
||||
|
||||
Tiny prompts (~1 sentence); per-call predicted cost printed + gated; usage/cost persisted (D30.10).
|
||||
Run: eval/.venv/bin/python eval/exp15/wirecheck.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from openai import OpenAI
|
||||
from dotenv import load_dotenv
|
||||
|
||||
EVAL = Path(__file__).resolve().parent.parent
|
||||
load_dotenv(EVAL / ".env")
|
||||
FACTS = json.load(open("/home/ubuntu/books/gu-zhenren/exp15/freeze_facts.json"))["prices"]
|
||||
OUT = Path("/home/ubuntu/books/gu-zhenren/exp15/wirecheck.json")
|
||||
PER_CALL_CAP = 0.05 # $ predicted per-call gate
|
||||
|
||||
PROMPT = [{"role": "system", "content": "Ты переводчик. Выведи ТОЛЬКО перевод."},
|
||||
{"role": "user", "content": "Переведи на русский: 方源笑了，握紧拳头。"}]
|
||||
|
||||
|
||||
def price(model, field):
|
||||
v = FACTS[model][field]
|
||||
return v[0] if isinstance(v, list) else v
|
||||
|
||||
|
||||
def cost(model, usage):
|
||||
pin, pout = price(model, "in"), price(model, "out")
|
||||
return (usage.get("prompt_tokens", 0) * pin + usage.get("completion_tokens", 0) * pout) / 1e6
|
||||
|
||||
|
||||
def reasoning_tokens(usage_obj):
|
||||
# xAI/OpenAI-style nested; be defensive across shapes
|
||||
d = usage_obj if isinstance(usage_obj, dict) else usage_obj.__dict__
|
||||
for k in ("reasoning_tokens",):
|
||||
if d.get(k):
|
||||
return d[k]
|
||||
ctd = d.get("completion_tokens_details")
|
||||
if ctd:
|
||||
ctd = ctd if isinstance(ctd, dict) else getattr(ctd, "__dict__", {})
|
||||
return ctd.get("reasoning_tokens", 0) or 0
|
||||
return 0
|
||||
|
||||
|
||||
def probe(name, base, env, model, extra=None, expect_reasoning=False):
|
||||
key = os.environ.get(env)
|
||||
if not key:
|
||||
return {"name": name, "error": f"no key {env}"}
|
||||
cli = OpenAI(base_url=base, api_key=key)
|
||||
kw = dict(model=model, messages=PROMPT, max_tokens=1200)
|
||||
if extra:
|
||||
kw.update(extra)
|
||||
try:
|
||||
r = cli.chat.completions.create(**kw)
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"name": name, "model": model, "error": f"{type(e).__name__}: {str(e)[:200]}"}
|
||||
u = r.usage
|
||||
usage = {"prompt_tokens": u.prompt_tokens, "completion_tokens": u.completion_tokens,
|
||||
"total_tokens": u.total_tokens}
|
||||
rt = reasoning_tokens(u)
|
||||
txt = (r.choices[0].message.content or "").strip()
|
||||
c = cost(model, usage)
|
||||
res = {"name": name, "model_requested": model, "model_returned": getattr(r, "model", None),
|
||||
"usage": usage, "reasoning_tokens": rt, "cost_usd": round(c, 6),
|
||||
"output_sample": txt[:120], "finish": r.choices[0].finish_reason}
|
||||
if expect_reasoning:
|
||||
res["reasoning_on_OK"] = rt > 0
|
||||
return res
|
||||
|
||||
|
||||
def main():
|
||||
checks = [
|
||||
# grok judge: NO reasoning_effort -> default (reasoning ON). Confirm reasoning tokens > 0.
|
||||
dict(name="grok-judge", base="https://api.x.ai/v1", env="XAI_API_KEY",
|
||||
model="grok-4.3", expect_reasoning=True),
|
||||
# mistral judge/arm: confirm slug resolves + version returned
|
||||
dict(name="mistral-large", base="https://api.mistral.ai/v1", env="MISTRAL_API_KEY",
|
||||
model="mistral-large-latest"),
|
||||
# glm-5 editor: confirm PAYG callable (thinking off for a clean quick probe)
|
||||
dict(name="glm-5-editor", base="https://api.z.ai/api/paas/v4", env="ZAI_API_KEY",
|
||||
model="glm-5", extra={"extra_body": {"thinking": {"type": "disabled"}}}),
|
||||
]
|
||||
results, total = [], 0.0
|
||||
for ch in checks:
|
||||
# per-call predicted cost gate (worst case max_tokens fully used as output)
|
||||
pred = price(ch["model"], "out") * 1200 / 1e6 + price(ch["model"], "in") * 60 / 1e6
|
||||
print(f"[gate] {ch['name']} ({ch['model']}) predicted <= ${pred:.4f} {'OK' if pred<=PER_CALL_CAP else 'BLOCKED'}")
|
||||
if pred > PER_CALL_CAP:
|
||||
results.append({"name": ch["name"], "error": f"per-call gate blocked (pred ${pred:.4f} > ${PER_CALL_CAP})"})
|
||||
continue
|
||||
res = probe(**ch)
|
||||
results.append(res)
|
||||
total += res.get("cost_usd", 0) or 0
|
||||
if "error" in res:
|
||||
print(f" ERROR: {res['error']}")
|
||||
else:
|
||||
extra = f" reasoning_tokens={res['reasoning_tokens']}" if res.get("reasoning_tokens") else ""
|
||||
print(f" OK model_returned={res['model_returned']} cost=${res['cost_usd']:.5f}{extra}")
|
||||
print(f" out: {res['output_sample']!r}")
|
||||
|
||||
OUT.write_text(json.dumps({"total_cost_usd": round(total, 6), "results": results}, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8")
|
||||
print(f"\ntotal wire-check spend: ${total:.5f} -> {OUT}")
|
||||
|
||||
# self-review asserts
|
||||
by = {r["name"]: r for r in results}
|
||||
grok = by.get("grok-judge", {})
|
||||
if "error" not in grok:
|
||||
assert grok.get("reasoning_on_OK"), f"grok reasoning NOT on (reasoning_tokens={grok.get('reasoning_tokens')})"
|
||||
print("OK: grok-4.3 judge reasoning-ON confirmed.")
|
||||
if "error" not in by.get("glm-5-editor", {}):
|
||||
print("OK: glm-5 PAYG callable with ZAI key.")
|
||||
if "error" not in by.get("mistral-large", {}):
|
||||
print(f"OK: mistral-large-latest -> {by['mistral-large'].get('model_returned')}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Loading…
Add table
Reference in a new issue