textmachine/backend/internal/pipeline/priceprojection.go

package pipeline

import (
	"strings"

	"textmachine/backend/internal/chunk"
	"textmachine/backend/internal/config"
	"textmachine/backend/internal/ledger"
	"textmachine/backend/internal/llm"
	"textmachine/backend/internal/text"
)

// priceprojection.go: what a book costs, said BEFORE it is bought (backlog row 278).
//
// ⛔ WHY IT EXISTS AND WHY IT IS NOT projectBookUSD. The engine already projects a book price
// (rebill.go), and that projection extrapolates from units it has ALREADY PAID FOR — it answers zero for
// a book that has never run, which is the only state the question is ever asked in. A buyer choosing how
// much to put on a book, and a platform choosing the ceiling to give the run, both need the number before
// the first call, so this one is derived from the TEXT, the engine's own calibration and its own prices.
// The two are complementary and neither replaces the other: this one is a priori, that one is measured.
//
// ⚠ TWO NUMBERS, TWO ARITHMETICS, AND THE SPLIT IS THE WHOLE DESIGN.
//
//   - `expected_usd` is what the run will be BILLED, and it is computed with the settlement formula
//     (ledger.CostUSD) over the engine's own fertility calibration — the same `est_out` coefficients the
//     chunker already sizes chunks with, which is the engine's existing belief about how much target text
//     a source produces. It is the answer to «what does this book cost».
//   - `step_max_usd` is the largest INDIVISIBLE RESERVATION, and it is computed with the reservation
//     formula (ledger.EstimateUSD) at the sizing the executor itself uses (baseMaxTokensFor,
//     maxTokensForAttempt). It is the answer to «what is the smallest ceiling under which this run can
//     move at all».
//
// Publishing one number for both would be wrong in a way that has already cost a book. A reservation is
// worst-case by construction (whole prompt uncached or cache-WRITTEN, completion running to max_tokens),
// so a projection built from reservations over-states the bill several times over and would have a
// platform refuse orders it can easily afford; a projection built from expected settlement understates
// the one quantity that actually refuses calls, which is exactly how a service arrived at «this book
// cannot be translated at any price». The wall is made of reservations, so the number about the wall is
// made of reservations — and the number about the bill is not.
//
// PAIR-AGNOSTIC BY CONSTRUCTION (общность §0.1): every coefficient here is data. The fertility is the
// pair's (`configs/pairs/*.yaml` → Pipeline.Segmentation.Fertility), the dense/sparse split is the
// engine's one sizing taxonomy (text.DenseScript), the prices and per-model floors are model data, and
// the stage list is the pipeline's. A pair that is not in this repository is priced by the same lines.

// unitPrice is the per-unit half of the projection.
type unitPrice struct {
	// SourceChars is the unit's ingested text in RUNES, spaces included — the figure a person can
	// reproduce in an editor, which is the whole reason it is runes and not bytes and not tokens.
	SourceChars int
	// Dense/Sparse split it by the engine's token-sizing taxonomy: a Han, kana or Hangul rune costs
	// roughly one model token, everything else roughly a third (text.DenseScript). Reported separately
	// because a reader converting chars to money needs to know which kind it has — the same chapter
	// count means very different money in the two.
	Dense, Sparse int
	// ExpectedUSD is what this unit is expected to be billed across every stage of the pipeline, one
	// attempt each. It excludes the book-level passes (see bookOnceUSD) and excludes retries, escalation
	// and repair — every one of those is CONDITIONAL, and folding a bad run's remedies into the headline
	// price would quote the exception as the rule.
	ExpectedUSD float64
	// PromptTokens is the source's size in the engine's own prompt-sizing taxonomy (EstimateTokens),
	// carried rather than recomputed: the step-max arithmetic needs exactly the number the expected-bill
	// arithmetic used, and two walks over one text are two chances to measure it differently.
	PromptTokens int
	// DraftTokens is the unit's draft as the EDIT wave will see it — the members' fertility-estimated
	// outputs joined. Carried for the same reason as PromptTokens: the step-max arithmetic must size the
	// edit call from exactly what the expected-bill arithmetic sized it from.
	DraftTokens int
}

// stagePrice is one pipeline stage resolved for pricing: its model's price, its template's token cost and
// its additive reasoning allowance. Resolved once per book — the stage list is static for a run.
type stagePrice struct {
	st        config.Stage
	price     ledger.ModelPrice
	tplTokens int
	reasoning int
	isDraft   bool
	// carriesSource says whether this stage's template puts the SOURCE text on the wire — the
	// `{{text}}` placeholder the engine already tracks for the echo-exposure warning
	// (runner.go/sourcePlaceholder). It is a property of the TEMPLATE, not of the stage's position: the
	// shipped editor is BILINGUAL (D30.1) and receives the source ALONGSIDE the draft, while a monolingual
	// editor arm does not. Reading it off the template rather than assuming «only the first stage sees the
	// source» is what keeps the projection right for both, and pair-agnostic besides.
	carriesSource bool
}

// pricePlan is the book's pricing inputs, resolved once. Nil-safe: a runner that could not resolve a
// template or a price yields no plan and the projection is simply absent from the manifest, which is the
// same degradation the manifest already takes for every other thing it cannot derive.
type pricePlan struct {
	stages []stagePrice
	// inject is the per-call injection allowance in tokens — the bank block's own budget, taken at its
	// WORST CASE because the bank the run will use does not exist yet at projection time (the manifest is
	// built before, and often long before, any bank is mined). Over-counting the prompt side is the safe
	// direction: the prompt is the cheap half of every call here.
	inject int
	// maxRegen is how many times a stage regenerates before escalating. It feeds step_max_usd and
	// NOTHING else — a regeneration is a remedy, not an expectation.
	maxRegen int
	// fertility is the PAIR's own output-token-per-source-char calibration, taken from the same
	// SegBudget the chunker cuts with. It is the one per-pair datum in this file and it is DATA: a pair
	// that is not in this repository brings its own coefficients and is priced by these same lines.
	fertility chunk.SegBudget
}

// pricePlanFor resolves the pricing inputs of this book's pipeline, or nil when it cannot.
func (r *Runner) pricePlanFor() *pricePlan {
	if r.Pricer == nil || len(r.Pipeline.Stages) == 0 {
		return nil
	}
	plan := &pricePlan{
		inject:    r.Pipeline.Context.GlossaryTokenBudget,
		maxRegen:  r.Pipeline.Retries.RegenerateBeforeEscalate,
		fertility: r.segBudget(),
	}
	if plan.maxRegen < 0 {
		plan.maxRegen = 0
	}
	for _, st := range r.Pipeline.Stages {
		sp := stagePrice{
			st:      st,
			price:   r.Pricer.PriceFor(st.ResolvedModel),
			isDraft: st.Role == roleTranslator,
			// The additive reasoning allowance is part of what the executor RESERVES (callEstimateUSD), so
			// it belongs to step_max_usd. It is deliberately not added to the expected bill: on the
			// providers that bill reasoning additively it is an allowance, not a forecast.
			reasoning: r.Models.AdditiveReasoningTokens(st.ResolvedModel, st.Reasoning, st.ReasoningMaxTokens),
		}
		// The template is on the wire of every call of this stage and it is not small — the shipped
		// editor prompt is over a thousand tokens. A projection that ignored it would understate the
		// prompt side of every unit in the book by the same amount, which is a bias rather than noise.
		//
		// ⚠ ASKED THROUGH THE EXECUTOR'S OWN FUNCTIONS, not by adding the fields up. SystemFor
		// (render.go) appends the few-shot block only when the stage's `few_shot` toggle is on, and the
		// shipped c1 editor has it OFF while its template still CARRIES the block — over a thousand
		// bytes. Summing System+FewShot+User would therefore over-charge every edit call of every book
		// running that arm, and it would do it silently, because both numbers look like numbers.
		if tpl := r.templates[st.Name]; tpl != nil {
			sp.tplTokens = EstimateTokens(tpl.SystemFor(fewShotEnabled(st))) + EstimateTokens(tpl.User)
			sp.carriesSource = strings.Contains(tpl.System+tpl.FewShot+tpl.User, sourcePlaceholder)
		}
		plan.stages = append(plan.stages, sp)
	}
	return plan
}

// projectUnit prices one output unit — every call the pipeline will make for it, at the granularity the
// EXECUTOR makes them.
//
// ⛔ THE DRAFT IS PER MEMBER CHUNK AND THE EDIT IS PER UNIT, and collapsing that was a bias in both
// directions at once. The wave driver fans the draft stage over CHUNKS (runDraftChunk, one render and one
// call per member) and the edit stage over UNITS, which a unit of several members makes visible twice:
// the template and the injection ride EVERY member's draft call, so charging them once per unit
// under-states the bill; and the draft's output budget is sized from ONE member, so sizing it from the
// whole unit over-states the largest single reservation — the very number a platform sets its minimum
// purchase by. On the shipped arm the second is hidden by the model floor and the first is a couple of
// percent; on an arm without a floor neither is.
//
// THE RECURSION IS THE EXECUTOR'S OWN. A stage sizes itself from the text it processes: the first stage
// from the source, every later stage from the previous stage's OUTPUT (stagerun.go, D2.5 — the editor
// works over the target-language draft, not over the source). So the projection walks the same chain,
// substituting the engine's fertility estimate of the draft for a draft that does not exist yet.
//
// ⚠ AN EDITING STAGE IS ASSUMED TO RETURN WHAT IT WAS GIVEN, in size. That is an assumption and it is
// named here rather than buried: an edit rewrites its input, it does not translate it again, so its
// output length tracks its input length. It is not a per-chapter constant — nothing here is calibrated
// per book — and it is the only place the chain is not read straight out of the executor.
func (p *pricePlan) projectUnit(u editUnit) unitPrice {
	src := u.sourceText()
	dense, sparse := text.DenseSparseCounts(src)
	up := unitPrice{SourceChars: len([]rune(src)), Dense: dense, Sparse: sparse, PromptTokens: EstimateTokens(src)}

	// The unit's draft, as the edit wave will see it: the members' outputs joined.
	draftTokens := 0
	for _, m := range u.Members {
		draftTokens += p.walkDraftCalls(m, func(sp stagePrice, srcTok, in, out int, inIsSource bool) {
			usage := llm.Usage{PromptTokens: p.promptTokens(sp, srcTok, in, inIsSource), CompletionTokens: out}
			up.ExpectedUSD += ledger.CostUSD(sp.price, usage)
		})
	}

	// The later stages, chained the same way — each reads what the previous produced and returns
	// something of that size.
	in, out := draftTokens, draftTokens
	laterReadsSource := false
	if draftTokens == 0 {
		// ⚠ A PIPELINE WITH NO TRANSLATOR STAGE AT ALL, and it needs TWO different fallbacks, not one.
		// runStageSequence starts such a wave with an empty `prev`, and the first stage's global index is
		// 0, so the executor sizes it from `ch.Text` — the SOURCE. That is its INPUT. Its OUTPUT is not
		// the source's token count, though: the stage is the one doing the translating, and what the
		// engine believes a source of this shape produces is the pair's fertility. Using the source count
		// for both under-states that stage's completion by the whole fertility coefficient — about a
		// sixth on the shipped pair.
		in, out = up.PromptTokens, p.expectedDraftTokens(dense, sparse)
		laterReadsSource = true // its FIRST stage reads ch.Text itself; the ones after it read a draft
	}
	for _, sp := range p.stages {
		if sp.isDraft {
			continue
		}
		usage := llm.Usage{PromptTokens: p.promptTokens(sp, up.PromptTokens, in, laterReadsSource), CompletionTokens: out}
		up.ExpectedUSD += ledger.CostUSD(sp.price, usage)
		in = out
		laterReadsSource = false
	}
	draftTokens = in
	up.DraftTokens = draftTokens
	return up
}

// walkDraftCalls visits every DRAFT call the executor will make for ONE member chunk, handing each the
// tokens that call actually carries, and returns the member's final draft size.
//
// ⛔ ONE WALK, TWO ARITHMETICS. The expected bill and the largest indivisible reservation are different
// questions about the SAME set of calls, and each answering it from its own loop is how two numbers that
// must agree stop agreeing — silently, because both look like numbers. It also encodes the chain the
// executor really runs: runStageSequence takes a chunk through EVERY draft stage in order, each reading
// the previous one's output, and nothing in the loader forbids a second translator stage (it validates
// stage ROLES and imposes no count — volume.go says so in its own words). Priced as a flat sum over
// stages instead, a second one would be charged against the SOURCE while the executor feeds it the first
// one's output.
func (p *pricePlan) walkDraftCalls(m chunk.Chunk, fn func(sp stagePrice, sourceTok, in, out int, inIsSource bool)) int {
	srcTok := EstimateTokens(m.Text)
	dense, sparse := text.DenseSparseCounts(m.Text)
	in, out := srcTok, p.expectedDraftTokens(dense, sparse)
	first, isSource := true, true
	for _, sp := range p.stages {
		if !sp.isDraft {
			continue
		}
		if !first {
			// A later draft stage reads what the previous one produced and — like an editing stage —
			// returns something of that size rather than translating again.
			in = out
		}
		fn(sp, srcTok, in, out, isSource)
		isSource = false
		first = false
	}
	if first {
		// ⛔ NO DRAFT STAGE RAN, SO THIS MEMBER PRODUCED NO DRAFT — and saying so is the whole point of
		// the zero. Returning `out` here regardless (which the first version did) made the caller's
		// translator-less fallback DEAD CODE: `draftTokens` came back non-zero, the branch that supplies
		// the two correct fallbacks never fired, and the first stage of such a pipeline was priced with a
		// draft-sized input it never receives. Found by a planting that SURVIVED — the mutation could not
		// change a branch nothing reaches.
		return 0
	}
	return out
}

// promptTokens is what one call of this stage puts on the wire, in tokens: its template, its input, the
// injection allowance — and the SOURCE, whenever the stage's template asks for it.
//
// ⛔ THE SOURCE IS THE PART THAT WAS MISSING, and its absence bent the number the expensive way. The
// executor renders EVERY stage with `RenderVars{Text: ch.Text, Draft: prev}` (stagerun.go), so a
// template holding `{{text}}` receives the source whatever its position in the pipeline — and the
// shipped editor is bilingual by ratified design (D30.1), which is the whole reason it can bind a canon
// to its source term. A projection that gave the later stages only the draft therefore dropped a
// source-sized block from the prompt of every edit call: on the shipped arm roughly a sixth of a unit's
// expected bill, and in the UNDER-stating direction, which is the one that lets a platform sell a book
// for less than it costs.
//
// It is read off the TEMPLATE and not off the stage's index, because that is where the fact lives: a
// monolingual editor arm carries no `{{text}}` and is priced without a source, on the same lines.
//
// ⚠ WHETHER THE INPUT *IS* THE SOURCE IS STATED BY THE CALLER, NOT INFERRED FROM EQUAL NUMBERS. The first
// version asked `inputTokens != sourceTokens` — and two token counts can coincide without being the same
// text: a pair whose fertility sits near 1.0 makes a draft the size of its source, and then the source
// silently stopped being charged to a stage that receives it (acceptance V2-8). Identity is a fact the
// caller has and the arithmetic does not.
func (p *pricePlan) promptTokens(sp stagePrice, sourceTokens, inputTokens int, inputIsSource bool) int {
	n := sp.tplTokens + inputTokens + p.inject
	if sp.carriesSource && !inputIsSource {
		// A stage whose input is already the source must not be charged for it twice.
		n += sourceTokens
	}
	return n
}

// expectedDraftTokens is the engine's own belief about how many target tokens a source of this shape
// produces — the fertility calibration the chunker already sizes every chunk with. Zero fertility (a
// pipeline that declares none) falls back to the sizing taxonomy, which is the same answer the chunker
// would give.
func (p *pricePlan) expectedDraftTokens(dense, sparse int) int {
	if p.fertility.FertCJK <= 0 && p.fertility.FertOther <= 0 {
		return dense + sparse/3
	}
	return int(p.fertility.EstOut(dense, sparse))
}

// projectBook is the WHOLE projection over one cut, in ONE pass: every unit's size and expected bill, the
// book roll-up, and the largest indivisible reservation.
//
// ⛔ ONE PASS AND ONE DERIVATION, and both halves are load-bearing. The first version of this file priced
// a unit inside the manifest loop and priced it AGAIN inside the book roll-up — two derivations of one
// number, which is the shape every drift in this repository has had — and walked the book's text seven
// times where three will do. The manifest is an accelerator bought with seconds on a 23 MB book; making
// it re-read the whole text twice more, to say the same thing twice, spends the saving on nothing.
//
// ⛔ TWO CALLERS, AND THEY MUST NEVER ANSWER DIFFERENTLY. The manifest writes this into the sidecar; a
// `status` that finds no current sidecar computes it from the cut it just made. A reader must get the
// same price either way — an accelerator that changes an ANSWER is a second source of truth — and the
// guarantee is held by there being one function rather than by two agreeing.
// TestManifestServesTheReadModelsIdentically is where a reader meets the consequence, and it caught
// exactly this: the projection first lived only on the manifest path, so `status` quoted a price or
// nothing depending on whether a sidecar happened to exist.
//
// Both results are nil when the pipeline cannot be priced at all; the fields are then simply absent,
// which is the degradation the manifest already takes for everything else it cannot derive.
func (r *Runner) projectBook(plan *pricePlan, units []editUnit) ([]unitPrice, *BookPrice) {
	if plan == nil {
		return nil, nil
	}
	prices := make([]unitPrice, len(units))
	bp := &BookPrice{BookOnceUSD: r.bookOnceUSD()}
	for i, u := range units {
		p := plan.projectUnit(u)
		prices[i] = p
		bp.SourceChars += p.SourceChars
		bp.ExpectedUSD += p.ExpectedUSD
		if step := r.stepMaxForUnit(plan, u, p); step > bp.StepMaxUSD {
			bp.StepMaxUSD = step
		}
	}
	// The book-level pass is a book-level charge, so it is added ONCE here rather than smeared over the
	// units: a book of three chapters pays the same consolidation as a book of three hundred, and
	// dividing it per unit is exactly the under-pricing of a short book that row 278 names.
	bp.ExpectedUSD += bp.BookOnceUSD
	return prices, bp
}

// stepMaxForUnit is the largest single reservation any call of ONE unit can ask for. The book's
// step_max_usd is the largest of these — the number a ceiling has to clear before the run can move at all.
//
// ⛔ RETRIES ARE IN IT, AND THAT IS THE POINT rather than caution. maxTokensForAttempt DOUBLES the output
// budget on every regeneration, so the second attempt of a stage reserves twice what the first did — and
// a refused RETRY does not degrade, it halts the book. A ceiling sized for attempt 0 therefore admits the
// first call of a unit and refuses its remedy forever: every resume replays the flagged attempt 0 for
// free and dies on attempt 1 again, so the book stops at a unit that can never finish while the run
// departs `paused` and asks for money that will not help.
//
// ⛔ THE OPTIONAL PASSES ARE NOT IN IT, for the mirror reason: the escalation hop, the repair sub-step
// and the terminology pass all DEGRADE when a ceiling refuses them (they catch errReserveCeiling and
// carry on), so a ceiling too small for them costs quality, not progress. Folding them in would raise the
// minimum purchase to pay for work the run is willing to skip.
func (r *Runner) stepMaxForUnit(p *pricePlan, u editUnit, up unitPrice) float64 {
	max := 0.0
	consider := func(sp stagePrice, sizing, prompt int) {
		base := r.baseMaxTokensFor(sp.st, sizing)
		for attempt := 0; attempt <= p.maxRegen; attempt++ {
			if usd := ledger.EstimateUSD(sp.price, prompt, maxTokensForAttempt(base, attempt), sp.reasoning); usd > max {
				max = usd
			}
		}
	}
	// PER MEMBER for the draft, because that is per CALL: a draft call is sized from ONE chunk, and sizing
	// it from the whole unit would inflate the largest single reservation by the member count — the number
	// a platform sets its minimum purchase by. The SAME walk the expected bill uses, so the two cannot
	// disagree about which calls exist or how they chain.
	for _, m := range u.Members {
		p.walkDraftCalls(m, func(sp stagePrice, srcTok, in, out int, inIsSource bool) {
			consider(sp, in, p.promptTokens(sp, srcTok, in, inIsSource))
		})
	}
	for _, sp := range p.stages {
		if sp.isDraft {
			continue
		}
		// A later stage sizes from the previous stage's OUTPUT (D2.5) — the unit's whole draft.
		consider(sp, up.DraftTokens, p.promptTokens(sp, up.PromptTokens, up.DraftTokens, false))
	}
	return max
}

// bookOnceUSD is the BOOK-level spend that is not a function of how many units are bought: the
// terminology consolidation and its classifier, which read the whole book's drafts once.
//
// ⚠ IT IS A BOUND AND NOT A FORECAST, and calling it anything else would be a lie the projection cannot
// support. The input of those passes is the candidate list the miner produces FROM THE DRAFTS, and the
// drafts do not exist when this number is published — so there is no honest way to predict what they will
// cost. What the engine CAN state is what it will never exceed: both passes are gated on their own
// configured budgets and the plan is trimmed against them before the first call is made. That bound is
// the useful figure anyway, because the question this projection is read to answer is «what ceiling does
// this book need», not «what will the invoice say».
//
// Zero when the passes are not configured to run at all — the field then says the truth, which is that
// this book has no book-level spend.
func (r *Runner) bookOnceUSD() float64 {
	g := r.Pipeline.Gates
	if !g.Terminology.Enabled || r.Pipeline.Mining.ContrastPath == "" {
		return 0
	}
	usd := g.Terminology.BudgetUSD
	// ⛔ THE CLASSIFIER'S BUDGET ONLY WHEN THE CLASSIFIER RUNS. `classify_types` gates the phase itself
	// (terminologist.go: `if !g.ClassifyTypes … return`), and the loader requires `classify_budget_usd > 0`
	// ONLY when that toggle is on — so with the phase off the key is free to hold whatever a config left
	// there, and adding it charged a book for a pass that cannot happen. Found by acceptance (V2-5).
	//
	// ⚠ WHERE IT BITES, STATED EXACTLY: the config this repository ships has the toggle ON
	// (`configs/pipeline-c1.yaml:170` = `classify_types: true`), so the shipped arm never showed the fault
	// — the budget it adds is a budget that is really spent. The defect reaches a configuration whose
	// toggle is OFF while the key still carries a figure left by an earlier edition, which the loader
	// permits precisely because it stops validating that key once the phase is off. ⚠ An earlier version
	// of this comment claimed the shipped arm was overcharged by half ($1.00 of $2.00); that was measured
	// nowhere and is false — the number is real, the arm is not.
	//
	// ⚠ AND THE TEST OF THIS FUNCTION PINNED THE WRONG NUMBER — a gate defending the defect, which no
	// review that reads green can catch.
	if g.Terminology.ClassifyTypes {
		usd += g.Terminology.ClassifyBudgetUSD
	}
	return usd
}