Land backend package 5: per-model max_tokens floor on primary and hop, postcheck base-dst union, budget-refusal and coverage-recall pins, D23.4 tails; golden byte-identical
This commit is contained in:
parent
73e5c0524c
commit
aa47e257c4
16 changed files with 585 additions and 19 deletions
|
|
@ -46,4 +46,4 @@ set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./intern
|
|||
|
||||
## Известный техдолг (не трогать молча — см. D-лог)
|
||||
|
||||
F3 at-most-once (после D15.2); Escal.Chains — валидируемый мёртвый конфиг (раннер читает только `escalate_to`, полные цепочки = шаг 7); D3-цепочка/канал B ЗАВЕДЕНЫ в `models.yaml` (пакет №3, слаги live-фактчекнуты 2026-07-10 — gemini ИМЕННО `-preview`); `escalation.budget_usd>0` требуется, чтобы single-hop `escalate_to` черновика стрелял (приёмка); min-budget Kimi≥16k/Gemini≥8k — комментарии, не схема; реализация content-addressed resume (D15.2) — после обязательных v3.1-правок (D22.2: langs→verdictSnapshotID, развязка cache_ttl).
|
||||
F3 at-most-once (после D15.2); Escal.Chains — валидируемый мёртвый конфиг (раннер читает только `escalate_to`, полные цепочки = шаг 7); D3-цепочка/канал B ЗАВЕДЕНЫ в `models.yaml` (пакет №3, слаги live-фактчекнуты 2026-07-10 — gemini ИМЕННО `-preview`); `escalation.budget_usd>0` требуется, чтобы single-hop `escalate_to` черновика стрелял (приёмка); per-model min-budget теперь СХЕМА (`capabilities.min_max_tokens`: deepseek 8000 / gemini 8000 / kimi 16000 — D24.3, флор по МОДЕЛИ вызова, у хопа своя; glm/grok без флора), не комментарий; реализация content-addressed resume (D15.2) — после обязательных v3.1-правок (D22.2: langs→verdictSnapshotID, развязка cache_ttl).
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import (
|
|||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
|
||||
"textmachine/backend/internal/llm"
|
||||
"textmachine/backend/internal/pipeline"
|
||||
|
|
@ -99,6 +100,32 @@ func TestRenderReportColumnsAndErrTail(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestErrTailTruncatesOnRuneBoundary pins the D23.4 fix: errTail carries raw provider
|
||||
// body snippets (CJK/Cyrillic), so a >120-byte tail must be cut on a RUNE boundary — a
|
||||
// bare byte slice s[:120] would split a multibyte rune into invalid UTF-8. The existing
|
||||
// column test does not exercise the boundary (its strings are short/ASCII), so a revert of
|
||||
// the fix survived it — this closes that gap. Mutation: replace the RuneStart walk-back in
|
||||
// errTail with `s = s[:120] + "…"` and this goes red.
|
||||
func TestErrTailTruncatesOnRuneBoundary(t *testing.T) {
|
||||
// One ASCII byte then 100 Cyrillic runes (2 bytes each): rune starts fall at offset 0
|
||||
// then the ODD offsets, so byte 120 (even) lands on a continuation byte — a mid-rune cut.
|
||||
long := "x" + strings.Repeat("я", 100) // 201 bytes, well over the 120 cap
|
||||
got := errTail("", long)
|
||||
if !utf8.ValidString(got) {
|
||||
t.Fatalf("errTail produced invalid UTF-8 — a byte-boundary cut split a rune: %q", got)
|
||||
}
|
||||
if strings.ContainsRune(got, '<27>') {
|
||||
t.Fatalf("errTail leaked a replacement char (broken rune): %q", got)
|
||||
}
|
||||
if !strings.HasSuffix(got, "…") {
|
||||
t.Fatalf("a truncated tail must end with the ellipsis, got %q", got)
|
||||
}
|
||||
// A short tail is returned verbatim (degraded + err joined, no truncation, no ellipsis).
|
||||
if s := errTail("cjk_artifact", "boom"); s != "cjk_artifact boom" {
|
||||
t.Fatalf("short tail must be returned verbatim, got %q", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRenderReportSections(t *testing.T) {
|
||||
var b strings.Builder
|
||||
flags := []store.ChunkStatus{
|
||||
|
|
|
|||
|
|
@ -115,6 +115,10 @@ models:
|
|||
deepseek-v4-flash:
|
||||
provider: deepseek
|
||||
price: { input_per_m: 0.14, cached_per_m: 0.0028, cache_write_per_m: 0, output_per_m: 0.28 }
|
||||
capabilities:
|
||||
# Floor per quirks (thinking ⊆ completion, subset): <8000 → reasoning eats the
|
||||
# budget, content пуст, finish=length (НЕ отказ). Schema, not a comment (D24.3).
|
||||
min_max_tokens: 8000
|
||||
note: >-
|
||||
Черновик (Р4). Автокэш по префиксу, отдельной цены записи нет.
|
||||
deepseek-chat/reasoner отключаются 24.07.2026 — сюда не возвращаться.
|
||||
|
|
@ -123,6 +127,10 @@ models:
|
|||
provider: deepseek
|
||||
# $0.435/$0.87, cache-hit $0.003625 (факт. 2026-07-10, api-docs.deepseek.com/quick_start/pricing).
|
||||
price: { input_per_m: 0.435, cached_per_m: 0.003625, cache_write_per_m: 0, output_per_m: 0.87 }
|
||||
capabilities:
|
||||
# Floor 8000 (D24.3): без него хоп наследовал бюджет праймари (2048/3291 на
|
||||
# приёмке этапа A) и вернул finish=length / 0 байт контента — эскалация впустую.
|
||||
min_max_tokens: 8000
|
||||
note: >-
|
||||
Эскалация черновика канала A, хоп 1 (D3): deepseek-эхо 25% на приёмочной 蛊真人 (D18) → single-hop
|
||||
фолбэк с draft. Наследует deepseek: thinking ON (эхо-мина того же класса — echoes_when_thinking_off),
|
||||
|
|
@ -157,10 +165,12 @@ models:
|
|||
provider: kimi
|
||||
price: { input_per_m: 0.95, cached_per_m: 0.16, cache_write_per_m: 0, output_per_m: 4.0 }
|
||||
# Альт-редактор (bake-off эксп-04, проиграл grok-4.3): принимает ТОЛЬКО temperature:1
|
||||
# (иначе 400), max_tokens≥16000 (иначе content пустой с finish=length), ~11k reasoning
|
||||
# tok/абзац. capabilities фиксируют квирк, чтобы edit-стадия на temp 0.4 не 400-ла.
|
||||
# (иначе 400), ~11k reasoning tok/абзац. capabilities фиксируют квирк, чтобы edit-стадия
|
||||
# на temp 0.4 не 400-ла; min_max_tokens закрепляет min-бюджет схемой (был комментарий).
|
||||
capabilities:
|
||||
temperature: { mode: force, value: 1 }
|
||||
# Floor 16000 (D24.3): <16000 → reasoning съедает лимит, content пустой, finish=length.
|
||||
min_max_tokens: 16000
|
||||
note: Альтернативный редактор (Р4).
|
||||
|
||||
grok-4.3:
|
||||
|
|
@ -190,8 +200,11 @@ models:
|
|||
# mandatory: thinking обязателен (thinking_budget:0 → HTTP 400, D6.3); адаптер молча глотает
|
||||
# reasoning:"off", forced thinking биллится как output. Учёт токенов — provider gemini reasoning:additive_total.
|
||||
reasoning: { control: mandatory }
|
||||
# Floor 8000 (D24.3): mandatory-thinking съедает бюджет; <8000 → перевод обрезан.
|
||||
# Раньше «дать max_tokens ≥8000» — комментарий; теперь схема (thinking ⊆ бюджета, additive_total).
|
||||
min_max_tokens: 8000
|
||||
note: >-
|
||||
Апекс канала A / судья Фазы 2 (D3). Дать max_tokens ≥8000 (thinking ⊆ бюджета → резерв не слепнет; settle
|
||||
Апекс канала A / судья Фазы 2 (D3). min_max_tokens 8000 держит thinking внутри бюджета (резерв не слепнет; settle
|
||||
биллит thinking через additive_total). Судья 18+ требует НАТИВНОГО Gemini (safety-off) — Ф2. Живьём 2026-07-10: chat 200.
|
||||
|
||||
mistral-large-2512:
|
||||
|
|
|
|||
|
|
@ -79,6 +79,11 @@ type CapabilitiesConfig struct {
|
|||
BudgetField string `yaml:"budget_field"` // max_tokens | max_completion_tokens
|
||||
Temperature *TemperatureCap `yaml:"temperature"`
|
||||
Reasoning *ReasoningCapCfg `yaml:"reasoning"`
|
||||
// MinMaxTokens is a per-model floor on the derived max_tokens (D24.3): a
|
||||
// thinking model whose reasoning eats the budget returns finish=length/empty
|
||||
// below this minimum. 0 (unset) = inherit / no floor. Schema, not a comment —
|
||||
// the Kimi≥16k / Gemini≥8k / DeepSeek≥8k min-budgets that were prose notes.
|
||||
MinMaxTokens int `yaml:"min_max_tokens"`
|
||||
}
|
||||
|
||||
// TemperatureCap declares how temperature reaches the wire.
|
||||
|
|
@ -276,6 +281,16 @@ func (m *Models) AdditiveReasoningTokens(modelName, reasoning string, declaredBu
|
|||
return 0
|
||||
}
|
||||
|
||||
// MinMaxTokens is the resolved per-model max_tokens FLOOR (D24.3), computed through
|
||||
// the same provider→model capability layering as the wire shape (ResolveCapability),
|
||||
// so what the runner floors against always matches what the snapshot folds. 0 = no
|
||||
// floor (reasoning-off models GLM/grok; an unknown model). The runner applies this to
|
||||
// the derived budget of EVERY call — primary attempts and the escalation hop take the
|
||||
// floor of THEIR own model — before the request_hash, so the floor is wire-visible.
|
||||
func (m *Models) MinMaxTokens(modelName string) int {
|
||||
return m.ResolveCapability(modelName).MinMaxTokens
|
||||
}
|
||||
|
||||
// APIKey resolves a provider's key from the environment ("" if the provider
|
||||
// declares no env var — the local backend).
|
||||
func (p Provider) APIKey() string {
|
||||
|
|
@ -396,6 +411,9 @@ func applyCapConfig(c *llm.Capability, cfg *CapabilitiesConfig) {
|
|||
OnExtraBody: cfg.Reasoning.OnExtraBody,
|
||||
}
|
||||
}
|
||||
if cfg.MinMaxTokens > 0 { // 0 = unset → inherit the provider/kind floor (model wins when it declares one)
|
||||
c.MinMaxTokens = cfg.MinMaxTokens
|
||||
}
|
||||
}
|
||||
|
||||
// validateCapabilities fail-fasts a capabilities block's enum fields (part of
|
||||
|
|
@ -404,6 +422,9 @@ func validateCapabilities(bad func(string, ...any), where string, c *Capabilitie
|
|||
if c == nil {
|
||||
return
|
||||
}
|
||||
if c.MinMaxTokens < 0 {
|
||||
bad("%s: capabilities.min_max_tokens must be ≥0 (a token floor), got %d", where, c.MinMaxTokens)
|
||||
}
|
||||
switch c.BudgetField {
|
||||
case "", "max_tokens", "max_completion_tokens":
|
||||
default:
|
||||
|
|
|
|||
|
|
@ -91,6 +91,20 @@ type Capability struct {
|
|||
Temp TempMode
|
||||
TempValue float64
|
||||
Reasoning ReasoningCap
|
||||
// MinMaxTokens is a per-model FLOOR on the derived max_tokens budget (D24.3):
|
||||
// a thinking model (DeepSeek/Gemini/Kimi) whose reasoning consumes the output
|
||||
// budget before any content is emitted returns finish=length or empty content
|
||||
// below a per-model minimum — acceptance stage A traced draft→deepseek-v4-pro
|
||||
// escalation hops receiving an inherited 2048/3291 and returning length. The
|
||||
// floor is NOT a wire body key (applyToBody never reads it); it is applied in
|
||||
// the runner's budget derivation BEFORE the request_hash, so it is wire-visible
|
||||
// via the resulting max_tokens. It lives on Capability so it is folded into the
|
||||
// job snapshot alongside the other wire-affecting inputs (snapshot.go marshals
|
||||
// the resolved Capability for the primary AND the escalate_to model) — editing
|
||||
// a floor is then a loud --resnapshot, not a silent false-hit. omitempty keeps a
|
||||
// floor-less model's snapshot byte-identical to before this field existed. 0 =
|
||||
// no floor (reasoning-off models: GLM, grok).
|
||||
MinMaxTokens int `json:",omitempty"`
|
||||
}
|
||||
|
||||
// withDefaults fills the OpenAI-compat baseline for zero fields: max_tokens +
|
||||
|
|
|
|||
|
|
@ -1,11 +1,14 @@
|
|||
package llm
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
|
@ -254,6 +257,45 @@ func TestRetryOn5xxThenSuccess(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestRetryLoopLogsWillRetryOnlyWhileAttemptsRemain pins the retryLoop logging contract
|
||||
// (D23.4, httpllm.go:87-89): a "will retry" WARN is emitted ONLY when an attempt actually
|
||||
// remains — carrying the REAL wait (retry_in), which Retry-After can silently stretch —
|
||||
// and NEVER before the terminal exhausted error. Before the fix the last failure also
|
||||
// logged "will retry" right before giving up. slog-capture, no live provider assertion.
|
||||
func TestRetryLoopLogsWillRetryOnlyWhileAttemptsRemain(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
logger := slog.New(slog.NewTextHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug}))
|
||||
var calls atomic.Int32
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls.Add(1)
|
||||
w.WriteHeader(http.StatusBadGateway) // always 5xx → retryable, runs to exhaustion
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
c := NewOpenAICompatClient(OpenAICompatConfig{Name: "prov", BaseURL: srv.URL, Profile: fastProfile()}, logger)
|
||||
_, err := c.Complete(context.Background(), LLMRequest{Model: "m", Messages: []Message{{Role: "user", Content: "u"}}, MaxTokens: 16})
|
||||
if err == nil {
|
||||
t.Fatal("a persistent 5xx must exhaust to an error")
|
||||
}
|
||||
if calls.Load() != 3 { // fastProfile MaxAttempts=3
|
||||
t.Fatalf("want 3 attempts, got %d", calls.Load())
|
||||
}
|
||||
logs := buf.String()
|
||||
// Exactly MaxAttempts-1 (2) retry warnings: after attempts 1 and 2, never after the
|
||||
// last one (the loop breaks on att+1>=MaxAttempts before the WARN).
|
||||
if n := strings.Count(logs, "attempt failed, will retry"); n != 2 {
|
||||
t.Fatalf("want exactly 2 'will retry' logs (none on the terminal attempt), got %d:\n%s", n, logs)
|
||||
}
|
||||
// Each retry log carries the real wait it is about to sleep.
|
||||
if !strings.Contains(logs, "retry_in=") {
|
||||
t.Fatalf("a retry log must carry retry_in (the actual wait, Retry-After-aware):\n%s", logs)
|
||||
}
|
||||
// The final outcome is the exhausted error, not another "will retry".
|
||||
if !strings.Contains(err.Error(), "exhausted") {
|
||||
t.Fatalf("terminal error must say exhausted, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBilledDecodeErrorOn2xxGarbage(t *testing.T) {
|
||||
// 2xx с нечитаемым телом: ретраится, а на исчерпании отдаёт типизированную
|
||||
// billed-ошибку — раннер по ней селтлит оценку вместо возврата резерва.
|
||||
|
|
|
|||
|
|
@ -240,8 +240,16 @@ func TestCoverageGateCatchesArtificialExcision(t *testing.T) {
|
|||
}
|
||||
}
|
||||
rate := float64(caught) / float64(total)
|
||||
// Acceptance bar (criterion 4, first half): the gate must catch ≥90% of the
|
||||
// artificial excisions.
|
||||
if rate < 0.90 {
|
||||
t.Fatalf("coverage gate caught %d/%d (%.0f%%) artificial excisions — below the ≥90%% acceptance bar", caught, total, rate*100)
|
||||
}
|
||||
t.Logf("mini-set: caught %d/%d (%.0f%%) artificial excisions", caught, total, rate*100)
|
||||
// Recall pinned (D24.1б): every drop fraction here exceeds 25%, so sent_cov falls
|
||||
// below 0.75 and ALL 12 excisions are caught — 100% recall, zero misses. A regression
|
||||
// that lets even one slip past drops this below total and fails loudly (not just a log).
|
||||
if caught != total {
|
||||
t.Fatalf("recall regressed: caught %d/%d (%.0f%%) — at these thresholds every >25%% excision must be caught (100%%)", caught, total, rate*100)
|
||||
}
|
||||
t.Logf("mini-set recall: caught %d/%d (100%%) artificial excisions (≥25%% drops, sent_cov<0.75)", caught, total)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,6 +13,22 @@ import (
|
|||
// один хоп на другую модель под budget_usd, результат РЕ-гейтится, retry-бюджет не
|
||||
// сбрасывается. Канал B (18+) Ф2 расширяет ИМЕННО этот файл (permissive-цепочки).
|
||||
|
||||
// applyModelFloor raises a derived max_tokens budget to the CALL model's per-model
|
||||
// floor (D24.3, capabilities.min_max_tokens): a thinking model returns finish=length
|
||||
// or empty content below a per-model minimum because reasoning consumes the budget
|
||||
// before content is emitted. The floor is taken by the model of THIS call — a primary
|
||||
// attempt floors against st.Model, an escalation hop against st.EscalateTo — so a hop
|
||||
// that inherits a smaller primary budget re-derives its own (exactly what broke the
|
||||
// draft→deepseek-v4-pro hops on acceptance stage A: they inherited 2048/3291 and
|
||||
// returned length). 0 floor (reasoning-off models) leaves the budget untouched.
|
||||
// Applied BEFORE the request_hash so it is wire-visible and snapshot-folded.
|
||||
func (r *Runner) applyModelFloor(base int, model string) int {
|
||||
if floor := r.Models.MinMaxTokens(model); floor > base {
|
||||
return floor
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
||||
// errReserveCeiling is wrapped into the error runAttempt returns when a book/day USD
|
||||
// ceiling denies a reservation. The PRIMARY path propagates it (the book durably
|
||||
// pauses and resumes once the ceiling is raised, D4); the OPTIONAL escalation hop
|
||||
|
|
@ -65,14 +81,20 @@ func (r *Runner) maybeEscalate(ctx context.Context, st config.Stage, snapID stri
|
|||
if !primary.cls.Reason.escalatable() || st.EscalateTo == "" {
|
||||
return out, nil
|
||||
}
|
||||
// The hop budget is the primary's base re-floored against the HOP's own model
|
||||
// (D24.3): the fallback may be a thinking model with a larger minimum than the
|
||||
// primary (draft deepseek-v4-flash → hop deepseek-v4-pro both floor 8000; a
|
||||
// floor-less grok primary → a floored hop lifts to the hop's minimum). This is
|
||||
// exactly the stage-A fix — the hop no longer inherits a sub-floor 2048/3291.
|
||||
hopMaxTokens := r.applyModelFloor(baseMaxTokens, st.EscalateTo)
|
||||
// Idempotency (self-review): if the fallback hop already happened its
|
||||
// checkpoint exists and was already paid — REPLAY it for free regardless of the
|
||||
// budget, so a crash AFTER the hop settled but BEFORE chunk_status was written
|
||||
// re-serves it on resume rather than discarding a paid, successful translation
|
||||
// and flipping the verdict OK→flagged. Only a FRESH hop is budget-gated. The
|
||||
// hash mirrors runAttempt's for (model=EscalateTo, attempt=0, maxTokens=base).
|
||||
// hash mirrors runAttempt's for (model=EscalateTo, attempt=0, maxTokens=hopMaxTokens).
|
||||
fbHash := RequestHash(r.Book.BookID, ch.Chapter, ch.ChunkIdx, 0, st.Name, st.Role, st.EscalateTo,
|
||||
st.Temperature, st.Reasoning, false, baseMaxTokens, snapID, msgs)
|
||||
st.Temperature, st.Reasoning, false, hopMaxTokens, snapID, msgs)
|
||||
fbExists, err := r.Store.GetCheckpoint(fbHash)
|
||||
if err != nil {
|
||||
return out, err
|
||||
|
|
@ -86,7 +108,7 @@ func (r *Runner) maybeEscalate(ctx context.Context, st config.Stage, snapID stri
|
|||
if !mayHop {
|
||||
return out, nil
|
||||
}
|
||||
fb, err := r.runAttempt(ctx, st, st.EscalateTo, snapID, ch, job, 0, baseMaxTokens, msgs, true)
|
||||
fb, err := r.runAttempt(ctx, st, st.EscalateTo, snapID, ch, job, 0, hopMaxTokens, msgs, true)
|
||||
if err != nil {
|
||||
// An OPTIONAL hop that trips a USD ceiling must NOT abort the whole book
|
||||
// (and re-abort on every resume): the chunk is already flagged, so keep
|
||||
|
|
|
|||
131
backend/internal/pipeline/escalation_budget_test.go
Normal file
131
backend/internal/pipeline/escalation_budget_test.go
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// escalation_budget_test.go closes the D24.1г acceptance caveat: escalation.budget_usd
|
||||
// was only exercised in the ">0 → the hop fires" direction; the REFUSAL by exhaustion was
|
||||
// never executed. Here a budget that a single hop overshoots must DENY the next flagged
|
||||
// chunk's hop — the flag stays honest, no fresh hop is made, and no escalation money is
|
||||
// spent beyond the first hop.
|
||||
|
||||
// setupTwoChapterEscalation writes a 2-chapter epub book whose draft escalates to
|
||||
// fake-fallback under budgetUSD. Two spine chapters → two chunks, both of which echo on
|
||||
// the primary (cjk_artifact) so both are escalation candidates.
|
||||
func setupTwoChapterEscalation(t *testing.T, providerURL string, budgetUSD float64) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
writeFile(t, filepath.Join(dir, "prompts", "translator.md"),
|
||||
"Переводи с {{source_lang}} на {{target_lang}}.\n---USER---\n{{text}}")
|
||||
writeFile(t, filepath.Join(dir, "prompts", "editor.md"),
|
||||
"Редактируй перевод.\n---USER---\nИсходник: {{text}}\nЧерновик перевода для редактуры: {{draft}}")
|
||||
writeFile(t, filepath.Join(dir, "models.yaml"), fmt.Sprintf(`
|
||||
prices_checked: %q
|
||||
default_model: fake-model
|
||||
providers:
|
||||
fake:
|
||||
kind: openai
|
||||
base_url: %q
|
||||
timeouts: { attempt_s: 5, max_attempts: 2, backoff_cap_s: 1 }
|
||||
models:
|
||||
fake-model:
|
||||
provider: fake
|
||||
price: { input_per_m: 1.0, cached_per_m: 0.1, cache_write_per_m: 0, output_per_m: 2.0 }
|
||||
fake-fallback:
|
||||
provider: fake
|
||||
price: { input_per_m: 1.0, cached_per_m: 0.1, cache_write_per_m: 0, output_per_m: 2.0 }
|
||||
`, time.Now().UTC().Format("2006-01-02"), providerURL))
|
||||
writeFile(t, filepath.Join(dir, "pipeline.yaml"), fmt.Sprintf(`
|
||||
core: C1
|
||||
version: 1
|
||||
defaults: { max_output_ratio: 2.0, min_max_tokens: 512 }
|
||||
retries: { regenerate_before_escalate: 0 }
|
||||
stages:
|
||||
- { name: draft, role: translator, model: fake-model, prompt: prompts/translator.md, prompt_version: v-test, temperature: 0.3, reasoning: "off", escalate_to: fake-fallback }
|
||||
- { name: edit, role: editor, model: fake-model, prompt: prompts/editor.md, prompt_version: v-test, temperature: 0.4, reasoning: "off" }
|
||||
escalation: { budget_usd: %g }
|
||||
`, budgetUSD))
|
||||
chapters := []epubChapter{
|
||||
{id: "c1", href: "ch1.xhtml", body: `<p>静かな図書館の朝。</p>`},
|
||||
{id: "c2", href: "ch2.xhtml", body: `<p>静かな図書館の夜。</p>`},
|
||||
}
|
||||
buildEPUBAt(t, filepath.Join(dir, "source.epub"), chapters, []string{"c1", "c2"})
|
||||
writeFile(t, filepath.Join(dir, "book.yaml"), `
|
||||
book_id: test-book
|
||||
title: Тест
|
||||
source_lang: ja
|
||||
target_lang: ru
|
||||
genre: ранобэ
|
||||
audience: тест
|
||||
venuti: 0.5
|
||||
honorifics: keep
|
||||
transcription: polivanov
|
||||
footnotes: minimal
|
||||
pipeline: pipeline.yaml
|
||||
models: models.yaml
|
||||
source_file: source.epub
|
||||
ceilings: { book_usd: 5.0, day_usd: 10.0 }
|
||||
`)
|
||||
return filepath.Join(dir, "book.yaml")
|
||||
}
|
||||
|
||||
// A budget that one hop overshoots denies the second flagged chunk's hop. The first chunk
|
||||
// escalates and resolves OK; the second stays honestly flagged, unescalated, with NO extra
|
||||
// escalation spend. Mutation: make escalationBudgetRemains always admit (drop `spent <
|
||||
// budget`) and the second chunk escalates → calls jump to 6 and it flips OK → red.
|
||||
func TestRunnerEscalationBudgetExhaustionDeniesLaterHop(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
srv := newJSONProvider(rec, echoOrClean) // primary echoes (cjk_artifact); fake-fallback cleans
|
||||
defer srv.Close()
|
||||
// budget 0.001 < one hop's cost (fakeCallUSD ≈ 0.00182): chunk1's hop is admitted
|
||||
// (spent 0 < budget) and overshoots to 0.00182; chunk2 then sees spent ≥ budget.
|
||||
bookPath := setupTwoChapterEscalation(t, srv.URL, 0.001)
|
||||
|
||||
r := newRunner(t, bookPath)
|
||||
defer r.Close()
|
||||
res, err := r.TranslateBook(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(res.Chunks) != 2 {
|
||||
t.Fatalf("want 2 chunks from 2 chapters, got %d", len(res.Chunks))
|
||||
}
|
||||
|
||||
// Calls: ch1 = primary + hop + edit (3); ch2 = primary only, hop denied, edit skipped (1).
|
||||
if rec.count() != 4 {
|
||||
t.Fatalf("want 4 provider calls (ch1: primary+hop+edit; ch2: primary only, hop denied), got %d", rec.count())
|
||||
}
|
||||
|
||||
ch1, ch2 := res.Chunks[0], res.Chunks[1]
|
||||
// Chapter 1: the first hop is admitted and resolves the chunk OK.
|
||||
if ch1.Disposition != DispOK || !ch1.Stages[0].Escalated {
|
||||
t.Fatalf("chapter 1 must escalate (budget still open) and resolve OK, got disp=%s escalated=%t", ch1.Disposition, ch1.Stages[0].Escalated)
|
||||
}
|
||||
// Chapter 2: the exhausted budget denies the hop — the flag stays, nothing escalated.
|
||||
if ch2.Disposition != DispFlagged || ch2.FlagReason != FlagCJKArtifact {
|
||||
t.Fatalf("chapter 2 must stay honestly flagged when the budget is exhausted, got disp=%s reason=%s", ch2.Disposition, ch2.FlagReason)
|
||||
}
|
||||
if ch2.Stages[0].Escalated {
|
||||
t.Fatalf("a budget-denied hop must NOT be recorded as an escalation, got %+v", ch2.Stages[0])
|
||||
}
|
||||
if ch2.Stages[1].Disposition != DispSkipped {
|
||||
t.Fatalf("chapter 2 edit must be skipped after the primary flag stands, got %+v", ch2.Stages[1])
|
||||
}
|
||||
if res.Flagged != 1 || res.ExitCode() != 2 {
|
||||
t.Fatalf("exactly the second chunk stays flagged, want flagged=1 exit=2, got flagged=%d exit=%d", res.Flagged, res.ExitCode())
|
||||
}
|
||||
|
||||
// Money: escalation spend is EXACTLY the one admitted hop — the denied hop cost nothing.
|
||||
esc, err := r.Store.EscalationSpentUSD("test-book")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if diff := esc - fakeCallUSD; diff > 1e-12 || diff < -1e-12 {
|
||||
t.Fatalf("escalation spend must be exactly one hop (%.6f), got %.6f — the denied hop must spend $0", fakeCallUSD, esc)
|
||||
}
|
||||
}
|
||||
|
|
@ -29,9 +29,16 @@ import (
|
|||
//
|
||||
// The fixture deliberately walks the money-relevant paths in one book: a 2-chunk
|
||||
// chapter (sticky window + injection), a spoiler-blocked term (since_ch: 2), a nested
|
||||
// glossary key (紋章 in 竜の紋章 → collision/AMBIGUOUS machinery), a CJK-echo draft
|
||||
// cured by the single-hop escalation re-gate (ch2), and a hard refusal where the
|
||||
// fallback also refuses → flag + skipped edit + exit 2 (ch3).
|
||||
// glossary key (紋章 inside 竜の紋章) exercising the collision/longest-match path, a
|
||||
// CJK-echo draft cured by the single-hop escalation re-gate (ch2), and a hard refusal
|
||||
// where the fallback also refuses → flag + skipped edit + exit 2 (ch3).
|
||||
//
|
||||
// D23.4 note: the nested key here resolves to CONFIRMED — the capture shows ambiguous=0
|
||||
// on every chunk (both keys are longest-match approved surfaces, not collision-prone
|
||||
// downgrades). A cell with AMBIGUOUS>0 (an auto/draft candidate forcing a post-check)
|
||||
// is an OPTIONAL fixture extension, deliberately NOT added in this pass: it would be a
|
||||
// fresh ratified golden capture of its own, and this package's re-capture is scoped to
|
||||
// the D24.3/D24.4 diff-class (floor max_tokens + post-check verdicts).
|
||||
//
|
||||
// Update (ONLY on a deliberate, ratified behaviour change — never to "fix" a red
|
||||
// refactor): TM_UPDATE_GOLDEN=1 go test ./internal/pipeline/ -run TestGolden
|
||||
|
|
|
|||
196
backend/internal/pipeline/maxtokens_floor_test.go
Normal file
196
backend/internal/pipeline/maxtokens_floor_test.go
Normal file
|
|
@ -0,0 +1,196 @@
|
|||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// maxtokens_floor_test.go pins the per-model max_tokens FLOOR (D24.3,
|
||||
// capabilities.min_max_tokens). The golden fixture uses only unfloored fake models,
|
||||
// so it does NOT exercise this behaviour — these dedicated tests do, mutation-verified:
|
||||
// - the floor lifts a small derived budget on the PRIMARY call (revert the stagerun
|
||||
// applyModelFloor line and TestMinMaxTokensFloorPrimary goes red);
|
||||
// - the escalation hop re-floors against ITS OWN model, not the inherited primary
|
||||
// budget (revert the maybeEscalate hopMaxTokens line and TestMinMaxTokensFloorHop
|
||||
// goes red — the exact stage-A bug where draft→deepseek-v4-pro hops inherited
|
||||
// 2048/3291 and returned finish=length);
|
||||
// - editing a floor is a loud --resnapshot (the floor is folded into the snapshot via
|
||||
// the resolved Capability), so TestMinMaxTokensFloorFoldedIntoSnapshot proves the
|
||||
// snapshotID moves when only the floor changes (invariant #2).
|
||||
|
||||
// setupFloorProject writes a ja→ru book whose draft model carries a per-model floor and,
|
||||
// optionally, escalates to a floored fallback. Both models sit on the same fake provider;
|
||||
// the mock distinguishes them by the "model" field. A tiny source keeps the derived base
|
||||
// budget (EstimateTokens × ratio, floored at the small global min_max_tokens) well BELOW
|
||||
// the per-model floors, so a wire max_tokens at the floor can only come from the floor.
|
||||
func setupFloorProject(t *testing.T, providerURL string, draftFloor, hopFloor int, withEscalation bool) string {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
writeFile(t, filepath.Join(dir, "prompts", "translator.md"),
|
||||
"Переводи с {{source_lang}} на {{target_lang}}.\n---USER---\n{{text}}")
|
||||
writeFile(t, filepath.Join(dir, "prompts", "editor.md"),
|
||||
"Редактируй перевод.\n---USER---\nИсходник: {{text}}\nЧерновик перевода для редактуры: {{draft}}")
|
||||
|
||||
draftCaps := ""
|
||||
if draftFloor > 0 {
|
||||
draftCaps = fmt.Sprintf("\n capabilities: { min_max_tokens: %d }", draftFloor)
|
||||
}
|
||||
hopCaps := ""
|
||||
if hopFloor > 0 {
|
||||
hopCaps = fmt.Sprintf("\n capabilities: { min_max_tokens: %d }", hopFloor)
|
||||
}
|
||||
writeFile(t, filepath.Join(dir, "models.yaml"), fmt.Sprintf(`
|
||||
prices_checked: %q
|
||||
default_model: fake-model
|
||||
providers:
|
||||
fake:
|
||||
kind: openai
|
||||
base_url: %q
|
||||
timeouts: { attempt_s: 5, max_attempts: 2, backoff_cap_s: 1 }
|
||||
models:
|
||||
fake-model:
|
||||
provider: fake
|
||||
price: { input_per_m: 1.0, cached_per_m: 0.1, cache_write_per_m: 0, output_per_m: 2.0 }%s
|
||||
fake-hop:
|
||||
provider: fake
|
||||
price: { input_per_m: 1.0, cached_per_m: 0.1, cache_write_per_m: 0, output_per_m: 2.0 }%s
|
||||
`, time.Now().UTC().Format("2006-01-02"), providerURL, draftCaps, hopCaps))
|
||||
|
||||
escLine, escBudget := "", ""
|
||||
if withEscalation {
|
||||
escLine = ", escalate_to: fake-hop"
|
||||
escBudget = "escalation: { budget_usd: 1.0 }"
|
||||
}
|
||||
// A small global min_max_tokens (512) is the floor the model-level floors must beat.
|
||||
writeFile(t, filepath.Join(dir, "pipeline.yaml"), fmt.Sprintf(`
|
||||
core: C1
|
||||
version: 1
|
||||
defaults: { max_output_ratio: 2.0, min_max_tokens: 512 }
|
||||
retries: { regenerate_before_escalate: 0 }
|
||||
stages:
|
||||
- { name: draft, role: translator, model: fake-model, prompt: prompts/translator.md, prompt_version: v-test, temperature: 0.3, reasoning: "off"%s }
|
||||
- { name: edit, role: editor, model: fake-model, prompt: prompts/editor.md, prompt_version: v-test, temperature: 0.4, reasoning: "off" }
|
||||
%s
|
||||
`, escLine, escBudget))
|
||||
|
||||
writeFile(t, filepath.Join(dir, "source.txt"), "静かな朝。")
|
||||
writeFile(t, filepath.Join(dir, "book.yaml"), `
|
||||
book_id: test-book
|
||||
title: Тест
|
||||
source_lang: ja
|
||||
target_lang: ru
|
||||
genre: ранобэ
|
||||
audience: тест
|
||||
venuti: 0.5
|
||||
honorifics: keep
|
||||
transcription: polivanov
|
||||
footnotes: minimal
|
||||
pipeline: pipeline.yaml
|
||||
models: models.yaml
|
||||
source_file: source.txt
|
||||
ceilings: { book_usd: 5.0, day_usd: 10.0 }
|
||||
`)
|
||||
return filepath.Join(dir, "book.yaml")
|
||||
}
|
||||
|
||||
// maxTokensForModel returns the max_tokens of the FIRST recorded body whose "model" is
|
||||
// modelName (the wire proof — the floor is applied before the request_hash, so what the
|
||||
// provider receives is the floored value).
|
||||
func maxTokensForModel(t *testing.T, rec *reqRec, modelName string) int {
|
||||
t.Helper()
|
||||
for _, body := range rec.all() {
|
||||
if strings.Contains(body, `"model":"`+modelName+`"`) {
|
||||
return maxTokensOf(t, body)
|
||||
}
|
||||
}
|
||||
t.Fatalf("no recorded request for model %q", modelName)
|
||||
return 0
|
||||
}
|
||||
|
||||
// The primary call's derived budget is lifted to the model's min_max_tokens floor.
|
||||
func TestMinMaxTokensFloorPrimary(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
srv := newJSONProvider(rec, draftEdit)
|
||||
defer srv.Close()
|
||||
bookPath := setupFloorProject(t, srv.URL, 8000, 0, false)
|
||||
|
||||
r := newRunner(t, bookPath)
|
||||
defer r.Close()
|
||||
if _, err := r.TranslateBook(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The tiny source derives far below 8000 (and below the 512 global floor too), so a
|
||||
// wire max_tokens of exactly 8000 can only be the per-model floor.
|
||||
if got := maxTokensForModel(t, rec, "fake-model"); got != 8000 {
|
||||
t.Fatalf("primary draft wire max_tokens = %d, want the model floor 8000 (D24.3)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The escalation hop is floored against ITS OWN model (fake-hop), NOT the smaller,
|
||||
// floor-less primary budget it inherits — the stage-A fix. The primary draft stays at the
|
||||
// small derived budget, proving the floor is per-call-model, not global.
|
||||
func TestMinMaxTokensFloorHop(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
srv := newJSONProvider(rec, func(body string) (string, string) {
|
||||
if isEditBody(body) {
|
||||
return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop"
|
||||
}
|
||||
if strings.Contains(body, `"model":"fake-hop"`) {
|
||||
return "Тихое утро.", "stop" // the hop translates cleanly → the re-gate passes
|
||||
}
|
||||
return "静かな図書館の朝。", "stop" // the primary echoes CJK → cjk_artifact → escalate
|
||||
})
|
||||
defer srv.Close()
|
||||
// draft floor 0 (primary keeps the small derived budget), hop floor 9000.
|
||||
bookPath := setupFloorProject(t, srv.URL, 0, 9000, true)
|
||||
|
||||
r := newRunner(t, bookPath)
|
||||
defer r.Close()
|
||||
res, err := r.TranslateBook(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !res.Chunks[0].Stages[0].Escalated {
|
||||
t.Fatalf("premise broken: the primary must escalate for the hop budget to be observable, got %+v", res.Chunks[0].Stages[0])
|
||||
}
|
||||
// The hop's own floor (9000) reaches the wire...
|
||||
if got := maxTokensForModel(t, rec, "fake-hop"); got != 9000 {
|
||||
t.Fatalf("escalation hop wire max_tokens = %d, want the HOP model floor 9000 — a hop must re-floor against its own model, not inherit the primary budget (D24.3)", got)
|
||||
}
|
||||
// ...while the floor-less primary stays at the small derived budget (global floor 512),
|
||||
// so the hop's 9000 is not merely an inherited value.
|
||||
if got := maxTokensForModel(t, rec, "fake-model"); got != 512 {
|
||||
t.Fatalf("primary draft wire max_tokens = %d, want the small derived/global budget 512 (proves the hop floor is per-call, not shared)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Editing a floor is a loud --resnapshot: the floor is folded into the snapshot via the
|
||||
// resolved Capability (snapshot.go marshals ResolveCapability for the stage AND its hop),
|
||||
// so a floor-only change must move the snapshotID — else a stale checkpoint false-hits a
|
||||
// diverged wire request (invariant #2 / D5.2).
|
||||
func TestMinMaxTokensFloorFoldedIntoSnapshot(t *testing.T) {
|
||||
rec := &reqRec{}
|
||||
srv := newJSONProvider(rec, draftEdit)
|
||||
defer srv.Close()
|
||||
|
||||
snapFor := func(draftFloor int) string {
|
||||
r := newRunner(t, setupFloorProject(t, srv.URL, draftFloor, 0, false))
|
||||
defer r.Close()
|
||||
id, _, err := r.snapshotID()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
none, f8000, f16000 := snapFor(0), snapFor(8000), snapFor(16000)
|
||||
if none == f8000 {
|
||||
t.Fatalf("adding a min_max_tokens floor must change the snapshotID (wire-affecting input, D24.3), got %s for both", none)
|
||||
}
|
||||
if f8000 == f16000 {
|
||||
t.Fatalf("changing the floor value must change the snapshotID, got %s for both", f8000)
|
||||
}
|
||||
}
|
||||
|
|
@ -156,6 +156,38 @@ func TestPostcheckWholeWordNoMasking(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestPostcheckBaseDstAlwaysChecked is the D24.4 repro for the acceptance-stage-A
|
||||
// nominative_gap (37/55 false misses): an entry whose decl.forms lists ONLY oblique
|
||||
// cases while the approved base dst appears in the NOMINATIVE in the output. The base
|
||||
// dst is itself an approved form, so it must ALWAYS be in the accepted set — not only as
|
||||
// a fallback when decl is empty. Mutation: revert dstFormPresent to the empty-decl
|
||||
// fallback (`forms := e.declForms; if len(forms)==0 {forms=[base]}`) and the positive
|
||||
// case below false-flags again → the test goes red.
|
||||
func TestPostcheckBaseDstAlwaysChecked(t *testing.T) {
|
||||
// 方源→Фан Юань, decl carries only oblique cases (the under-seeded reality of stage A).
|
||||
fang := gl("方源", "Фан Юань", "", "approved")
|
||||
fang.Decl = declJSON(false, "Фан Юаня", "Фан Юаню", "Фан Юанем") // oblique-only, NO nominative base
|
||||
b := bankFrom([]store.GlossaryEntry{fang})
|
||||
sel := b.Select("方源走了。", 1, nil, 0)
|
||||
if len(sel.injected) != 1 {
|
||||
t.Fatalf("expected 方源 injected, got %d", len(sel.injected))
|
||||
}
|
||||
// Positive (the fix): the nominative base present but ABSENT from the oblique-only
|
||||
// decl → must NOT be a miss now that the base is always accepted.
|
||||
if m := b.postcheck(sel.injected, "На поляну вышел Фан Юань, окружённый гу."); len(m) != 0 {
|
||||
t.Fatalf("nominative base dst present yet flagged — the base form must always be accepted (D24.4 nominative_gap): %v", m)
|
||||
}
|
||||
// The oblique decl forms still work (decl-awareness retained, not replaced).
|
||||
if m := b.postcheck(sel.injected, "Все боялись Фан Юаня."); len(m) != 0 {
|
||||
t.Fatalf("oblique decl form present yet flagged: %v", m)
|
||||
}
|
||||
// Negative (precision preserved): neither the base nor any decl form present → a real
|
||||
// miss. The union only ADDS an accepted form; it must not blind a genuine drop.
|
||||
if m := b.postcheck(sel.injected, "На поляну никто не вышел."); !hasMiss(m, "方源") {
|
||||
t.Fatalf("dst and all forms absent must still flag a miss (union must not blind precision): %v", m)
|
||||
}
|
||||
}
|
||||
|
||||
// TestE1PostcheckCatchesPoisoning: with complete decl, a WRONG rendering (poisoning /
|
||||
// the model ignoring the glossary) is still a true-positive flag — the post-check's
|
||||
// actual job. Precision does not come for free: the decl forms are inflections of the
|
||||
|
|
|
|||
|
|
@ -79,15 +79,35 @@ func (b *MemoryBank) postcheck(injected []pickedEntry, output string) []postchec
|
|||
// substring. The whole-word rule closes self-review #3: a DROPPED short translit name
|
||||
// («Ван») must not be masked by an unrelated common word that merely contains its letters
|
||||
// («караВАН», «диВАН», «ИВАН» — a different person). Decl-aware (the research/14
|
||||
// requirement): the stored decl forms carry the inflections, so precision and recall are
|
||||
// both satisfied when decl is complete; the base-dst fallback (empty decl) is the naive
|
||||
// case the E1 measurement quantifies. noutRunes is the normalized output pre-decomposed.
|
||||
// requirement): the stored decl forms carry the inflections a boundary regexp would miss.
|
||||
//
|
||||
// The accepted set is the UNION of the base dst AND the stored decl forms (D24.3/D24.4):
|
||||
// the base translation is itself an approved form, so it is ALWAYS checked — not only as a
|
||||
// fallback when decl is empty. Checking it only on empty decl false-flagged 37/55
|
||||
// acceptance-stage-A misses (nominative_gap): an approved dst present in the NOMINATIVE
|
||||
// while `decl.forms` listed only oblique cases (方源→Фан Юань present in the output, decl
|
||||
// carried only genitive/dative). Requiring the base form to be duplicated into decl is
|
||||
// redundant input every future book would trip on.
|
||||
//
|
||||
// PRECISION/RECALL TRADEOFF, not a free win (adversarial review, pkg 5): enlarging the
|
||||
// accepted set is monotonic — it can only turn misses into passes, never the reverse. On
|
||||
// the measured stage-A data those removed misses are all FALSE positives on correctly
|
||||
// rendered nominatives (precision up, the 37/55). The one way it can COST recall is narrow
|
||||
// but real: an entry whose decl OMITS the nominative AND whose base dst is a common word
|
||||
// that recurs elsewhere while the term itself was DROPPED for its firing occurrence
|
||||
// (剑→меч dropped as «клинок», yet an unrelated «меч» sits elsewhere) — this bag-of-words
|
||||
// presence check then passes and masks the drop. That class is rare on real data (translit
|
||||
// names do not recur coincidentally) and TOLERABLE only while this stays a FLAGGER: a
|
||||
// hard-gate promotion (postcheck_gate) MUST re-measure recall on common-noun terms, not
|
||||
// assume it — the "recall unaffected" phrasing in the D24.4 rationale is imprecise here.
|
||||
// The remaining inflection_gap (18/55: a plural rendered but only the singular seeded) is a
|
||||
// SEED completeness fix (Polygon), not a code one. noutRunes is the normalized output
|
||||
// pre-decomposed.
|
||||
func dstFormPresent(e *memoryEntry, noutRunes []rune) bool {
|
||||
forms := e.declForms
|
||||
if len(forms) == 0 {
|
||||
forms = []string{normalizeTargetForm(e.dst)}
|
||||
if base := normalizeTargetForm(e.dst); base != "" && containsWholeWord(noutRunes, []rune(base)) {
|
||||
return true
|
||||
}
|
||||
for _, f := range forms {
|
||||
for _, f := range e.declForms {
|
||||
if f != "" && containsWholeWord(noutRunes, []rune(f)) {
|
||||
return true
|
||||
}
|
||||
|
|
|
|||
|
|
@ -90,6 +90,12 @@ func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, sn
|
|||
if baseMaxTokens < r.Pipeline.Defaults.MinMaxTokens {
|
||||
baseMaxTokens = r.Pipeline.Defaults.MinMaxTokens
|
||||
}
|
||||
// Per-model floor (D24.3): a thinking model needs a minimum budget or its
|
||||
// reasoning truncates the content (finish=length / empty). Applied to the
|
||||
// PRIMARY model here; the escalation hop re-floors against ITS own model
|
||||
// (maybeEscalate) — the fix for stage-A's draft→deepseek-v4-pro hops that
|
||||
// inherited 2048/3291 and returned length.
|
||||
baseMaxTokens = r.applyModelFloor(baseMaxTokens, st.Model)
|
||||
|
||||
maxRegen := r.Pipeline.Retries.RegenerateBeforeEscalate
|
||||
if maxRegen < 0 {
|
||||
|
|
|
|||
|
|
@ -69,7 +69,7 @@ func TestKillMinus9LosesAtMostOneCall(t *testing.T) {
|
|||
}
|
||||
path := filepath.Join(t.TempDir(), "kill9.db")
|
||||
|
||||
for round := 0; round < 3; round++ {
|
||||
for round := 0; round < 5; round++ { // D23.4: 3→5 rounds — more SIGKILL windows per run
|
||||
cmd := exec.Command(exe, "-test.run", "^TestHelperKillLoop$", "-test.v")
|
||||
cmd.Env = append(os.Environ(), killLoopEnv+"="+path)
|
||||
if err := cmd.Start(); err != nil {
|
||||
|
|
|
|||
|
|
@ -772,6 +772,33 @@ keep-alive (Ф1–2), инъекция глоссария (`selective`), пор
|
|||
3. **Пинг (зона docs/):** `architecture/06-memory-risk-registry.md:71` цитирует «snapshotID (runner.go:145-163)» и `07-strategic-review` содержит runner.go-линки — после декомпозиции указатели ведут в сетап-файл; поправить при следующем касании доков (сами механизмы не менялись: snapshotID/memoryVersion → snapshot.go).
|
||||
4. Известная info-грань: OpenReadOnly на read-only ФС (бэкап-снапшот) не работает (WAL требует создать -shm) и создаёт -shm/-wal рядом с БД при чтении голой копии; хинт ошибки «пустая/битая база?» в этом кейсе вводит в заблуждение. Не регресс (старый путь падал раньше и жёстче); лечить — только формулировкой хинта, авто-`immutable=1` небезопасен при живом писателе.
|
||||
|
||||
### 2026-07-11 (пакет-5) — Фиксы приёмки этапа A (D24.3/24.4) + тесты-оговорки + D23.4-хвосты ЗАВЕРШЕНО
|
||||
|
||||
По `BACKEND_PACKAGE5_SESSION_PROMPT.md`. **Ничего не коммичено** (лендинг — оркестратор). `go build ./... && go vet ./... && go test -race ./...` — зелёные. Все новые тесты mutation-verified (реверт фикса валит именно его тест — подтверждено исполнением). Финал — агентское адверсариальное ревью диффа (4 оси find→refute-by-default verify, author≠reviewer): 1 CONFIRMED minor (неточность коммента о recall у union — закрыт, см. оговорку 4), 0 critical/major. Осведомлён о D26-заморозке этапа B: пакет №5 остаётся в очереди (line-4 CURRENT-STATE) и гейтит этап B; фиксы валидны независимо от editor-диагностики exp12.
|
||||
|
||||
**Сделано:**
|
||||
|
||||
1. **Per-model floor `min_max_tokens` (D24.3) — смена wire-поведения.** Поле в схему `capabilities` (`config.CapabilitiesConfig.MinMaxTokens`), резолвится через `ResolveCapability` (провайдер→модель) и живёт в `llm.Capability` (тег `json:",omitempty"`) → **фолдится в snapshot автоматически** и для primary (`ResolveCapability(st.Model)`), и для hop (`ResolveCapability(st.EscalateTo)`) — правка флора = громкий `--resnapshot`. Применение: `Runner.applyModelFloor(base, model)` ДО request_hash — на праймари (`stagerun.go`, по `st.Model`) И на хопе (`escalation.go` `maybeEscalate`, по `st.EscalateTo`: `hopMaxTokens` в fbHash И в runAttempt — ровно фикс этапа A, где хоп наследовал 2048/3291 → finish=length). Флоры в `models.yaml`: deepseek-v4-flash/pro **8000**, gemini-3.1-pro-preview **8000**, kimi-k2.6 **16000**; glm-5/5.1/grok-4.3 — без поля. Комментарии-заглушки «дать ≥8000/16000» → схема. Валидация `min_max_tokens<0`. README-техдолг обновлён (комментарий→схема). Флор не течёт в тело как ключ (`applyToBody` его не читает) — только через max_tokens; резерв EstimateUSD растёт на флорнутых вызовах, committed по факту не меняется.
|
||||
|
||||
2. **Post-check `dstFormPresent`: базовый dst всегда проверяется (D24.4) — смена вердиктов.** Множество проверяемых форм = `{normalizeTargetForm(dst)} ∪ declForms` ВСЕГДА (было: база — только фолбэк при пустом decl). Закрывает 37/55 ложных промахов этапа A (nominative_gap: approved-dst в именительном, decl только косвенные). Union монотонен (только промах→пасс) → precision↑; poisoning ловится по-прежнему (тест). recall на измеренных данных не страдает, НО строго это precision/recall-трейдофф с узким классом ложных пропусков — см. оговорку 4. inflection_gap (18/55, мн.ч.) остаётся сид-фиксом полигона.
|
||||
|
||||
3. **Golden re-capture — дифф-класс ПУСТОЙ (byte-identical, sha256 совпал, git diff пуст).** Прогон `TM_UPDATE_GOLDEN=1` не изменил `capture.golden`. Причина (проверено чтением фикстуры): (а) фикстура использует ТОЛЬКО нефлорнутые `fake-model`/`fake-fallback` → floor 0 → нет wire-изменений; `omitempty` держит Capability-JSON нефлорнутых моделей байт-в-байт → snapshot неизменен; (б) в сиде каждый entry с непустым decl уже несёт базовый dst первой формой, а два записанных промаха (紋章→герб ch1/0; 竜の紋章→Драконья печать ch2/0) НЕ имеют в проверяемом выходе ни базовой формы, ни decl-формы → union не спасает ни один → вердикты неизменны. Пустой дифф ⊆ санкционированных классов {(a) max_tokens флорнутых, (b) postcheck-вердикты}. Оба новых поведения покрыты выделенными mutation-verified юнит-тестами (golden их не упражняет — нет флорнутых моделей/oblique-only-сида в фикстуре).
|
||||
|
||||
4. **Тесты (все новые mutation-verified; числа):**
|
||||
- `TestMinMaxTokensFloorPrimary/Hop/FoldedIntoSnapshot` (`maxtokens_floor_test.go`, новый): primary флор на wire (max_tokens==8000); хоп re-флорится по СВОЕЙ модели (primary=512, hop=9000 — доказывает per-call, не global); правка флора двигает snapshotID. Мутации: снять primary-флор → 512≠8000 FAIL; хоп наследует базу → 512≠9000 FAIL.
|
||||
- `TestPostcheckBaseDstAlwaysChecked` (`memory_e1_test.go`): repro этапа A (方源→Фан Юань, oblique-only decl + именительный в выходе → промаха НЕТ) + негатив (dst и все формы отсутствуют → промах ЕСТЬ). Мутация: реверт к empty-decl-фолбэку → положительный кейс валится с `[{方源 Фан Юань confirmed}]`.
|
||||
- `TestRunnerEscalationBudgetExhaustionDeniesLaterHop` (`escalation_budget_test.go`, новый; D24.1г): 2-главный epub, оба чанка эхают, budget 0.001 < 1 хоп. Гл.1 хоп допущен (spent 0<budget) → OK; гл.2 хоп ДЕНИЕД (spent≥budget) → флаг остаётся, edit skip, escalation-spend ровно 1 хоп (денег на деньед-хоп $0), 4 вызова, exit 2. Мутация: `spent<budget+1e9` → гл.2 эскалирует, 6 вызовов FAIL.
|
||||
- `TestCoverageGateCatchesArtificialExcision` (D24.1б, предсуществовал — пакет coverage-гейта): синтетические пропуски (drop 30/40/50/60% предложений × zh/ja/en) → recall **12/12 = 100%**, контроль 0 ложных. Усилил: recall-число теперь ПИННУЕТСЯ ассертом (`caught==total`), не только логом; ≥90%-бар приёмки сохранён.
|
||||
- `TestErrTailTruncatesOnRuneBoundary` (`render_test.go`; D23.4): >120 байт с континьюейшн-байтом на офсете 120 → валидный UTF-8, без U+FFFD, суффикс «…». Мутация: `!RuneStart(...)&&false` → `\xd1…` невалид FAIL (реверт фикса раньше выживал сьют — гэп закрыт).
|
||||
- `TestRetryLoopLogsWillRetryOnlyWhileAttemptsRemain` (`httpllm_test.go`; D23.4, slog-capture): персистентный 5xx, MaxAttempts=3 → ровно 2 «will retry» (не на последней попытке), несут `retry_in`, финал — «exhausted». Мутация: снять `att+1>=MaxAttempts break` → 3 «will retry» FAIL.
|
||||
- kill9 rounds 3→5 (`kill9_test.go`, D23.4). Комментарий golden_test.go:32 смягчён (AMBIGUOUS-ячейка: выбрано смягчение коммента, НЕ правка фикстуры — держит golden байт-в-байт; ambiguous=0 в капче задокументирован, AMBIGUOUS>0-ячейка = опц. расширение фикстуры отдельной ратификацией).
|
||||
|
||||
**Оговорки / вопросы оркестратору:**
|
||||
1. **Golden не упражняет новые поведения** (флор/union) — фикстура нефлорнута и сид уже полон базой-в-decl. Покрытие — выделенными тестами. Рекомендация (опц., отдельная ратификация): при следующем осознанном golden-рефреше добавить в фикстуру (i) стадию на флорнутой модели и (ii) entry с oblique-only decl + именительным в выходе — тогда golden запинит и wire-max_tokens флора, и union-вердикт. Не делал сейчас: это правка фикстуры = golden-дифф класса ВНЕ {a,b} санкции пакета.
|
||||
2. **Реальный `configs/models.yaml` теперь даёт другой snapshot** для deepseek/gemini/kimi (флор в Capability). Ожидаемо и совпадает с планом D24.5 «фиксы → единый resnapshot → этап B»; книга gu-zhenren вне git, resnapshot на её стороне. Тесты boevoy-конфига (echo_mine) зелёные.
|
||||
3. **Порядок floor vs global MinMaxTokens:** оба — max(), порядок неважен; флор только ПОВЫШАЕТ (0 не понижает). Хоп: `max(унаследованная_база_праймари, флор_хопа)` — наследование бюджета сохранено (мандат), флор добавлен сверху.
|
||||
4. **⚠ Пинг (зона docs/, формулировка D24.4): «recall не страдает» у union-фикса — НЕТОЧНО.** Адверсариальное ревью пакета (4 оси find→verify, 1 CONFIRMED minor) поймало: расширение accepted-set монотонно (только промах→пасс), это precision/recall ТРЕЙДОФФ, не бесплатный выигрыш. На данных этапа A убранные промахи — все ложные (precision↑, 37/55). Но есть узкий РЕАЛЬНЫЙ класс потери recall: entry, у которого decl БЕЗ именительного И база — нарицательное слово, которое встречается в выходе отдельно, тогда как сам термин для своего вхождения ДРОПНУТ (`剑→меч` дропнут как «клинок», но «меч» стоит рядом не по делу) → bag-of-words-проверка пасует, дроп замаскирован. На реальных данных редко (транслит-имена не совпадают случайно) и ТЕРПИМО пока это ФЛАГГЕР. **Если `postcheck_gate` когда-либо флипается в hard-gate — обязателен пере-замер recall на нарицательных терминах (не предполагать).** Код НЕ менял (union = ратифицированный D24.4); поправил только собственный коммент в `mempostcheck.go` на честный трейдофф + предупреждение о промоушене. Поведение/golden не тронуты. Рекомендация: в D24.4-логе смягчить «recall не страдает» → «на измеренных данных recall не страдает; hard-gate требует пере-замера».
|
||||
|
||||
## Полигон
|
||||
(секция параллельной сессии — записи добавлять сюда)
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue