textmachine/backend/internal/pipeline/paidtail.go

package pipeline

import (
	"encoding/json"
	"fmt"

	"textmachine/backend/internal/llm"
	"textmachine/backend/internal/store"
)

// paidtail.go: WHAT THE MONEY BOUGHT, decomposed — the largest single line of the first paid run and the
// one axis no $0 surface showed.
//
// Measured on the cold run of 31.08: the money that did not become shipped text was the biggest item in
// the run, larger than escalation, and both `status` and `report` published only a TOTAL. An operator was
// told what a book cost and never what part of it bought nothing (disclosure law §2.1).
//
// ⛔ WHY THIS IS NOT THE OBVIOUS QUERY, and the reason is measured rather than argued. The obvious slice is
// `request_log WHERE ok = 0 AND cost_usd > 0`. On that run it selects 11 rows and $0.12316089 — and THREE
// different things:
//   - the classifier's three calls ($0.00918827), which SUCCEEDED. Their replies are healthy term tables
//     (`三转蛊师<TAB>term …`, finish=stop), flagged because the CJK-echo exemption covered ONLY the
//     terminologist AT THE TIME — it covers both bank roles now (terminologist.go, `isBankRole`) — so a
//     bilingual table read as an echo of its own source;
//   - one editor call ($0.01865424) whose text SHIPPED after a cosmetic sanitizer strip;
//   - and the rest, which genuinely bought nothing.
// A surface built on that slice would have told the operator he lost 29.2% more than he did — a money
// figure with a false cause, which is the disease this pack is about, committed by its own cure.
//
// SO THE DECOMPOSITION IS BUILT FROM THE CHECKPOINTS, not from the verdict column. Checkpoints are the
// money (`committed == SUM(checkpoints)` is an invariant of the ledger), they are append-only, and their
// INSERTION ORDER is the only thing that distinguishes a superseded call from the one that stands — which
// is why CheckpointUsageForBook is documented as ordering by rowid and `attempt` explicitly cannot do it.
//
// FOUR classes, exhaustive over paid checkpoints, and each states something literally true:
//   - SUPERSEDED — a later paid call for the SAME position replaced it. Retries and escalation hops both
//     land here, from the same rule, with no need to read either column.
//   - BANK — it bought the book's TERMINOLOGY rather than a chunk of its text.
//   - WITHHELD — the position's last paid call produced nothing shippable (its row carries no final hash).
//   - SHIPPED — it bought the bytes a reader gets.
// Nothing here re-derives money: every figure is a sum of costs the ledger already settled.
//
// ⛔ THE BANK CLASS EXISTS BECAUSE ITS ABSENCE MADE THIS FILE COMMIT THE DEFECT IT WAS BUILT AGAINST, and
// the acceptance caught it. The bank roles are checkpointed under a SYNTHETIC stage at chapter 0 and write
// NO chunk_status row at all — deliberately, they are not a wave — so `shipped` can never be true for
// them and the whole terminology contour fell into WITHHELD. On the cold run that is $0.04980482 of a
// $0.133 «loss», i.e. the surface overstated by 59.8%: worse than the 29.2% overstatement this file's own
// header rejects the naive `ok = 0` slice for. A live probe on a HEALTHY book printed «$0.010920 (85.7%)
// did NOT become shipped text» about a glossary pass that worked perfectly.
//
// The class is identified by the STAGE, which is the addressing label those calls carry
// (terminologyStageName). The REPAIR role is deliberately NOT in it: repair checkpoints carry their
// stage's real name, so they map to a chunk_status position like any other call and are classified by
// whether that position shipped — which is right for the repair that STANDS, and what it costs a chunk
// that needed several is spelled out below.
//
// ⛔ AND THE BANK ROLES NEED THE ROLE COLUMN TOO, which is why `pos` below is not the chunk_status triple
// for them. A chunk stage is addressed by (chapter, chunk, stage), and a retry or an escalation hop at
// that address is genuinely competing to produce the SAME bytes: the later one replaces the earlier, which
// is what `superseded` means. (⚠ The repair role shares that address WITHOUT sharing that property — a
// repair rewrites a bounded span, not the stage's whole output, and several repairs of one chunk buy
// provably DISJOINT spans while landing on one triple. So they file each other as superseded today. That
// is a defect of the same family as the one fixed below, it is NOT fixed here — a second money class in
// one landing — and it is what TestTheRoleSplitDoesNotMoveARepairsMoney pins as unmoved rather than as
// right.) The bank roles are a third case again: they share ONE synthetic stage at
// chapter 0 and number their BATCHES in chunk_idx, so the classifier's batch 0 and the terminologist's
// batch 0 arrive at the same triple having bought two different things, and the earlier of the two was
// reported as money that bought nothing. Measured on the run of 08.09: three ledger rows
// (`0 0 terminology classifier` · `0 0 terminology terminologist` · `0 1 terminology terminologist`)
// printed as «the book's TERMINOLOGY $0.004595 (2)» — the classifier's $0.009013, the largest of the
// three, filed under `superseded` while it was bought and used.
//
// The role is read for the bank stage ALONE and not folded into every position, and folding it in would
// not have been the cure anyway: it separates a repair from the stage call it follows — a different class,
// on a different row, moved unannounced — while leaving repairs of ONE chunk on top of each other, since
// every one of them carries the same role and only the span tells them apart.
//
// ⚠ A superseded BANK call is still superseded, and that is not a special case but the point: the whole
// contour is re-bought whenever the drafted set grows (backlog row 233), and that money genuinely bought
// nothing the second time. Only the call that STANDS is the bank's product.

// PaidTail is the decomposition of a book's committed spend by WHAT IT BOUGHT.
type PaidTail struct {
	// ShippedUSD bought the text that ships. SupersededUSD was replaced by a later paid call for the same
	// position. WithheldUSD bought a position that produced nothing shippable.
	ShippedUSD    float64 `json:"shipped_usd"`
	SupersededUSD float64 `json:"superseded_usd"`
	WithheldUSD   float64 `json:"withheld_usd"`
	// BankUSD bought the book's TERMINOLOGY — the consolidated glossary the editor injects — rather than a
	// chunk of its text. It is a product, not a loss, and it is separated because the bank roles write no
	// chunk_status row, so the "did it ship" question is not merely false for them: it is undefined.
	BankUSD float64 `json:"bank_usd"`
	// The call counts behind the money — a single expensive call and twenty cheap ones are different
	// problems with the same dollar figure.
	ShippedCalls    int `json:"shipped_calls"`
	SupersededCalls int `json:"superseded_calls"`
	WithheldCalls   int `json:"withheld_calls"`
	BankCalls       int `json:"bank_calls"`
	// TotalUSD is the sum of the three, i.e. every paid checkpoint of the book. It is published so a
	// reader can check the decomposition against the ledger's own committed figure instead of trusting it.
	TotalUSD float64 `json:"total_usd"`
	// WorstPosition names the single position that spent the most on money that did not ship, so the
	// largest loss is a place an operator can go to rather than a number. Empty when nothing was lost.
	WorstPosition string  `json:"worst_position,omitempty"`
	WorstUSD      float64 `json:"worst_usd,omitempty"`
}

// LostUSD is the money that bought NOTHING — neither shipped text nor bank. It excludes BankUSD by
// construction: a glossary pass that worked is not a loss, and printing it as one is the exact shape of
// false-cause this whole pack exists to remove.
func (t PaidTail) LostUSD() float64 { return t.SupersededUSD + t.WithheldUSD }

// paidTail decomposes the book's paid checkpoints. Pure over its inputs and free of I/O so the
// classification can be tested without a database behind it.
//
// `usage` MUST be in insertion order (CheckpointUsageForBook guarantees rowid order) — the whole
// superseded/standing distinction is that order, and a caller that sorts it differently gets a different
// and wrong answer.
func paidTail(usage []store.CheckpointUsage, statuses []store.ChunkStatus) PaidTail {
	shipped := make(map[pos]bool, len(statuses))
	for _, cs := range statuses {
		// A position ships when its row carries the hash the read models resolve text through. A flagged
		// row carries none, and neither does a skipped one. chunk_status has no role column and needs
		// none: only the bank roles share a stage, and they write no chunk_status row at all.
		shipped[pos{chapter: cs.Chapter, chunkIdx: cs.ChunkIdx, stage: cs.Stage}] = cs.FinalHash != ""
	}
	// The LAST paid call at each position is the one that stands; every earlier one was replaced.
	lastPaid := map[pos]int{}
	for i, u := range usage {
		if u.CostUSD <= 0 {
			continue // derived $0 projections (banknote/sanitized exports) are not calls and cost nothing
		}
		lastPaid[posOf(u)] = i
	}
	var t PaidTail
	lost := map[pos]float64{}
	for i, u := range usage {
		if u.CostUSD <= 0 {
			continue
		}
		p := posOf(u)
		t.TotalUSD += u.CostUSD
		switch {
		case lastPaid[p] != i:
			t.SupersededUSD += u.CostUSD
			t.SupersededCalls++
			lost[p] += u.CostUSD
		case u.Stage == terminologyStageName:
			// The call that STANDS for a bank position bought the bank. It has no chunk_status row to
			// have shipped, and asking «did it ship» of it is a category error, not a failure.
			t.BankUSD += u.CostUSD
			t.BankCalls++
		case shipped[p]:
			t.ShippedUSD += u.CostUSD
			t.ShippedCalls++
		default:
			t.WithheldUSD += u.CostUSD
			t.WithheldCalls++
			lost[p] += u.CostUSD
		}
	}
	// Deterministic worst-position: ties break on the position, never on map order.
	for p, v := range lost {
		name := p.name()
		if v > t.WorstUSD || (v == t.WorstUSD && name < t.WorstPosition) {
			t.WorstUSD, t.WorstPosition = v, name
		}
	}
	return t
}

// pos is what a paid call BOUGHT, as an identity: two calls at the same pos are two purchases of one
// thing, and the later one replaces the earlier.
type pos struct {
	chapter, chunkIdx int
	stage             string
	// role is set for the BANK stage alone — see the file header. Empty everywhere else, so a chunk
	// stage's calls keep colliding exactly as they must.
	role string
}

// posOf addresses one billed call. The bank roles are the only calls whose stage is not their address.
func posOf(u store.CheckpointUsage) pos {
	p := pos{chapter: u.Chapter, chunkIdx: u.ChunkIdx, stage: u.Stage}
	if u.Stage == terminologyStageName {
		p.role = u.Role
	}
	return p
}

// name is the operator-facing address. Chapter 0 is the BOOK level (the bank roles address their batches
// there), so it is spelled as such rather than as a chapter nobody has, and the role is spelled with it:
// two positions that print one name would send an operator to the wrong one of them.
func (p pos) name() string {
	if p.chapter == 0 {
		s := fmt.Sprintf("book/batch%d/%s", p.chunkIdx, p.stage)
		if p.role != "" {
			s += "/" + p.role
		}
		return s
	}
	return fmt.Sprintf("ch%d/chunk%d/%s", p.chapter, p.chunkIdx, p.stage)
}

// estimatedSpend is how much of a book's committed money is an ESTIMATE rather than a figure the
// provider reported, and over how many calls. It sits beside paidTail because it asks the same kind of
// question of the same rows: that one decomposes committed spend by what it BOUGHT, this one by how
// well it is KNOWN.
//
// It is the engine's half of PD-441 and the condition attached to D39.230 п.1. Without it the platform
// bills committed_usd and can say «at least this much», never «at least X, up to Y».
//
// ⛔ THE TEST IS «PAID WITH NO TOKENS TO SHOW FOR IT», not a list of causes. Cost is derived from usage
// everywhere else in the engine, so a settled row with zero tokens and a non-zero cost can only have
// come from a reservation estimate — whatever ended the call: a body that would not decode, a deadline
// of ours, a stopped run, a broken socket, a paid 2xx that reported no usage. A list of finish_reasons
// would answer the same question today and quietly stop answering it the day a sixth way to pay
// without a token count is added.
//
// Reading the checkpoints rather than request_log is what makes the pair a decomposition of
// committed_usd instead of a second, telemetry-shaped opinion about it: the two are the same money, and
// a resumed run writes no new estimate row while its money stands.
//
// ⛔ `committed == SUM(checkpoints)` HOLDS ONLY UNTIL A REDRIVE, and the difference is money this
// figure would otherwise disown. A redrive DELETES the checkpoints of the stages it re-attacks and
// leaves their spend committed — «after a redrive committed(spend) >= SUM(checkpoints), the safe
// direction» (store/chunkstatus.go). Derived from the surviving rows alone, the published share then
// falls to zero and tells the platform that money nobody can account for was measured. So the gap is
// carried INTO the estimate by the same rule the rows are: we hold a cost and have no token count to
// justify it. Measured on a live redrive: committed unchanged at $0.001056, estimated dropped to $0.
func estimatedSpend(usage []store.CheckpointUsage, committedUSD float64) (rows int, usd float64) {
	var accounted float64
	for _, u := range usage {
		accounted += u.CostUSD
	}
	for _, u := range usage {
		if u.CostUSD <= 0 {
			continue // derived $0 checkpoints are not calls and cost nothing
		}
		var tok llm.Usage
		if err := json.Unmarshal([]byte(u.UsageJSON), &tok); err != nil {
			// Unreadable usage on a paid row is the same state as absent usage: a cost we cannot
			// justify from tokens. Counting it as measured publishes the more comfortable answer.
			rows, usd = rows+1, usd+u.CostUSD
			continue
		}
		if tok.PromptTokens == 0 && tok.CompletionTokens == 0 && tok.ReasoningTokens == 0 {
			rows, usd = rows+1, usd+u.CostUSD
		}
	}
	// Committed money with no checkpoint behind it at all: a redrive threw the evidence away and kept
	// the spend. It is counted as ONE more unaccounted line rather than as a per-call figure, because
	// how many calls it stood for is exactly what was deleted. The float slack keeps a sum of prices
	// from inventing a nanodollar of «unaccounted».
	if gap := committedUSD - accounted; gap > 1e-9 {
		rows, usd = rows+1, usd+gap
	}
	return rows, usd
}