textmachine/backend/internal/pipeline/rebill.go
2026-09-15 14:18:58 +03:00

570 lines
34 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

package pipeline
import (
"context"
"fmt"
"textmachine/backend/internal/chunk"
"textmachine/backend/internal/store"
)
// rebill.go: CONSENT TO A RE-PAYMENT (D20.2-Q2, spec §7.3 of backend/docs/D15.2-*.md) — the Р6
// "consent to a CONCRETE spend" gate standing in front of every path that would bill again for work
// this book has already been billed for.
//
// GRANULARITY (ratified with this pack). The spec's §7 projection is PER-CHUNK and rides `guard_hash`,
// which does not exist in this engine at all: drift here is BOOLEAN (a stored row's snapshot either is
// or is not the current one) and a snapshot move misses EVERY checkpoint of its wave. So consent is
// built at the granularity the engine actually re-pays at — the SNAPSHOT — and the projection is the
// honest reading of that: "the units already paid for under a superseded snapshot will be paid for
// again; their recorded tokens at TODAY'S price table come to $X" (reprice.go, row 181 — until then it
// summed the historical cost_usd, which is a different table the moment a vendor moves its prices).
//
// Two consequences of that granularity, both deliberate:
// - the spec's §7.1-бис editor-cascade fix ("a stage strictly below a re-billed unit is itself
// re-billed") is DEGENERATE here — everything of the moved wave is re-billed already, so the
// cascade rule adds nothing. Its ratified ordering ("the cascade lands BEFORE the flag semantics")
// is therefore not violated but inapplicable;
// - the spec's EstimateUSD fallback ("for cascade units with no past price") has no subject either:
// every unit counted here HAS stored usage, because it is counted precisely for having been paid.
// A paid row that re-prices to $0 is a genuinely $0 model (local/priced-zero), so 0 is its honest
// contribution rather than a gap to estimate around.
//
// WHAT IS NOT PROJECTED (documented, not silent): the CONTENT axis. A source edit that leaves the chunk
// manifest intact moves a chunk's content_hash but not the snapshot, so it re-bills exactly the touched
// chunks and is invisible here — the same caveat status.bookChunks carries, and the half that belongs
// to content-addressed resume (v3), not to this gate.
// Ratified default threshold (D20.2-Q2): `min($0.50, 5% × ProjectedBookUSD)`, with the $0.50 acting as
// an absolute FLOOR when the book has no processed units yet (the 5% branch would then be $0 and would
// demand consent for a one-cent append). A book may override it with `rebill_consent_usd`.
const (
rebillConsentFloorUSD = 0.50
rebillConsentShare = 0.05
)
// RebillConsent is the operator's answer to `--accept-rebill[=usd]`: the Р6 consent to a CONCRETE
// spend. A bare flag accepts whatever the projection turns out to be; `--accept-rebill=1.50` accepts it
// only while it stays at or below $1.50, so the consent names an amount rather than a blanket.
type RebillConsent struct {
Given bool // the flag was passed at all
Capped bool // a ceiling was named (--accept-rebill=<usd>)
CapUSD float64 // that ceiling; meaningful only when Capped
}
// RebillProjection is what a run would re-pay: the chunk×stage units resolved under a superseded
// snapshot, and what buying them again would cost AT THE CURRENT PRICE TABLE (reprice.go).
type RebillProjection struct {
Rows int
USD float64
// HistoricalRows counts the units inside USD that could not be fully re-priced and carry the amount
// they were billed at instead: a billed-decode row with no usage on file, a row whose calls no longer
// account for its money, or one whose newest calls are provably not its own (reprice.go). It exists so
// the operator-facing text can name what the number is made of rather than claim a re-pricing it did
// not achieve.
HistoricalRows int
// ModelMovedRows counts the units inside USD whose stage resolves today to a model other than the one
// their calls were SENT to (reprice.go — the comparison is against the requested slug, so a provider
// canonicalising its own name is not a move): they are priced at the dearer of the two, and the token
// count the new model would spend is unknowable — so the larger this count, the more USD is an upper
// ESTIMATE rather than a re-pricing of known tokens. It is the number the consent text names when it
// says so (D39.150 п.1: the caveat travels with the figure).
ModelMovedRows int
// OutputUnits is the same re-payment counted in OUTPUT UNITS — the granularity the manifest publishes
// as units_total, that `--max-units` bounds, and that the platform sells chapters in.
//
// ⚠ IT EXISTS BECAUSE `Rows` IS A DIFFERENT UNIT WEARING THE SAME WORD. Rows counts chunk×stage, the
// BILLING unit; every other "units" number the engine puts on the wire (total_units, progress totals,
// chapters[].units_total) is the OUTPUT unit. Measured on a three-chapter fixture, one document carried
// `rebill_units: 15` two lines under `total_units: 6` with nothing marking the difference — and the
// ratio is not a constant a consumer could divide out: it is len(Members)·nDraftStages + nEditStages,
// which varies unit by unit WITHIN one book and whose factors never cross the seam at all. A platform
// sizing a re-pass from the estimate and passing that number to --max-units would buy several times the
// book it meant to, which is this pack's own defect reproduced one layer down.
OutputUnits int
// Repinned counts the units the run will serve for $0 despite a moved snapshot (pack-20 point 5): the
// move was bank-only and their rendered bytes are unchanged. It is not part of the amount — it is the
// number that makes the amount believable, because before pack-20 every one of these was counted as a
// re-payment and the operator was asked to consent to a whole wave for one signed word.
Repinned int
}
// projectRebill sums the units that a run under the CURRENT config would pay for a second time.
//
// A unit counts when all three hold: it carries a snapshot (a row written by a real run), it was BILLED
// (ok/flagged — a `skipped` row never reached a provider and cost nothing), and its snapshot differs
// from what its own WAVE renders now. The per-wave comparison is load-bearing: a bank-mining enrichment
// moves only the edit-wave snapshot, and projecting the draft wave against it would report the whole
// book as re-billed when the drafts in fact resume at $0 (the "re-paid ONCE" invariant).
//
// Two kinds of stored row are ORPHANS — they exist but nothing will run them again, so counting them
// would ask the operator to consent to money that will not be spent:
// - a row whose STAGE the current pipeline no longer has (a renamed/retired stage);
// - a row whose CHUNK the current manifest no longer has (the source was shortened, so the position
// is gone). Both waves address their rows at manifest chunk positions — a draft row at its own
// chunk, an edit row at its unit's leader chunk — so one membership test covers both.
//
// Over-estimating is the safe direction for a consent gate, but a number the operator is asked to
// approve and then not charged is exactly what makes such a number stop being read. Which is why the
// amount is not a safety margin either way: it is the stored tokens re-priced through the same seam the
// reservation will use, so the two answer the same question with the same table.
//
// CONTENT AWARENESS (pack-20 point 5, closing the phase-1 P3 finding). Snapshot divergence alone
// over-counts: since the bank is folded per wave, signing ONE term marked the whole edit wave as re-billed
// even though almost every unit's rendered bytes are unchanged. The projection now models the resume
// predicate the run actually applies (repin.go): a bank-only snapshot move whose unit renders the SAME
// content hash resumes at $0 and is counted as re-pinned, not re-paid. What remains in the amount is the
// units where the changed term genuinely occurs — the "N units, ~$X" the owner asked to see before
// consenting. The reproduction is $0 (string rendering + store reads, no provider), and a position whose
// inputs cannot be reproduced falls back to the conservative answer: counted as a re-payment.
//
// The wave snapshots are rendered LAZILY, so a draft-only pipeline never renders an edit-wave snapshot
// it has no stages for.
//
// `manifest` supplies only the POSITIONS still in the book (the membership test above), which is why it
// may be the text-free projection of the persisted manifest (manifest.go). `withText` supplies the same
// chunks WITH their source, and is called only on the content-aware branch below — the one that has to
// re-render injected bytes and therefore genuinely needs the text. Splitting the two is what lets a
// status read skip a 1.4 s re-chunk it almost never needs, without ever computing a content hash over an
// empty string.
func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Chunk, withText func() ([]chunk.Chunk, error), rp *repricer) (RebillProjection, error) {
var p RebillProjection
decider := newRepinDecider(r)
var contentHashes map[chunkKey]map[string]string
live := make(map[chunkKey]bool, len(manifest))
for _, ch := range manifest {
live[chunkKey{ch.Chapter, ch.ChunkIdx}] = true
}
// Which OUTPUT unit each position belongs to, so the same re-payment can be reported in the unit the
// seam sells in as well as the one it bills in. An edit row lives at its unit's leader chunk and a
// leader is one of the unit's members, so this one map covers both waves.
leaderOf := make(map[chunkKey]chunkKey, len(manifest))
for _, u := range r.outputUnits(manifest) {
leader := chunkKey{u.Chapter, u.FirstChunkIdx}
for _, m := range u.Members {
leaderOf[chunkKey{m.Chapter, m.ChunkIdx}] = leader
}
}
// ONE lazy reproduction of the rendered content hashes, shared by both branches that need them. It was
// inline in the bank-only branch; the source-edit check needs the same map, and a second construction
// site is how the two would come to disagree about what «the bytes this run would render» means.
reproduce := func() (map[chunkKey]map[string]string, error) {
if contentHashes != nil {
return contentHashes, nil
}
full, ferr := withText()
if ferr != nil {
return nil, fmt.Errorf("pipeline: re-chunk the source for the re-bill content check: %w", ferr)
}
contentHashes = r.cachedRenderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget))
return contentHashes, nil
}
touched := map[chunkKey]bool{}
draftNames := stageNameSet(r.waveStagesIndexed(waveDraft))
editNames := stageNameSet(r.waveStagesIndexed(waveEdit))
rendered := map[wave]string{}
current := func(w wave) (string, error) {
if s, ok := rendered[w]; ok {
return s, nil
}
s, _, err := r.snapshotIDForWave(w)
if err != nil {
return "", fmt.Errorf("pipeline: render the current snapshot for the re-bill projection: %w", err)
}
rendered[w] = s
return s, nil
}
for _, cs := range statuses {
if cs.SnapshotID == "" || cs.Disposition == string(DispSkipped) {
continue
}
if !live[chunkKey{cs.Chapter, cs.ChunkIdx}] {
continue // the position is gone from the manifest — nothing will re-run it
}
var w wave
switch {
case draftNames[cs.Stage]:
w = waveDraft
case editNames[cs.Stage]:
w = waveEdit
default:
continue // a stage the current pipeline does not run is never re-billed
}
cur, err := current(w)
if err != nil {
return p, err
}
if cs.SnapshotID == cur {
// ⛔ THE SNAPSHOT MATCHING IS NOT ENOUGH, and this branch used to stop here (backlog row 238).
// The source is deliberately NOT in the snapshot, so an edit to the source file moves the
// row's rendered CONTENT while its snapshot id stays identical — and the run's own resume
// fast-path checks exactly that (`cs.ContentHash == contentHash`, stagerun.go) and re-buys the
// unit. The projection said «$0 resume» and `translate` charged: money silent about work it
// was about to do (disclosure law §2.1). The docstring above claimed this function «models the
// resume predicate the run actually applies» — it modelled it in the bank-only branch only,
// which made the claim itself an instance of §2.2.
//
// The check is CONDITIONAL on a cheap probe rather than always-on, and that is deliberate: the
// content hashes cost a re-chunk of the source (~1.4 s on a 23 MB book) that `status` almost
// never needs, and the split exists to let a status read skip it. sourceMovedUnderTheRows
// answers «could any rendered byte have moved» from the stored manifest's validity key, which
// folds the source SHA — so the expensive answer is only bought once there is a question.
if !r.sourceMovedUnderTheRows() {
// Nothing here will be RE-PAID for: the probe says no input under these rows moved, which is
// the question this projection answers. ⚠ It is not the same as «resumes at $0» — a stop mark
// is re-attacked and bought on the next run — but that purchase is not a RE-payment and does
// not belong in this figure.
continue
}
hashes, herr := reproduce()
if herr != nil {
return p, herr
}
if want, ok := hashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && want == cs.ContentHash {
continue // the source moved somewhere, but not under THIS row: still a $0 resume
}
// Either the rendered bytes differ or they cannot be reproduced. Both fall through to the
// conservative answer — the same direction the rest of this function takes — and the row is
// counted as a re-payment, which is what the run will actually do.
}
// The snapshot moved — but that is not yet a re-payment. Ask the two questions the resume path asks.
bankOnly, berr := decider.bankOnlyMove(cs.SnapshotID, w)
if berr != nil {
return p, berr
}
if bankOnly {
if contentHashes == nil { // rendered once, lazily: a book with no bank move never pays for it
if _, ferr := reproduce(); ferr != nil {
return p, ferr
}
}
if h, ok := contentHashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && h == cs.ContentHash {
p.Repinned++
continue // the bank moved somewhere else in the book; this unit is re-pinned for $0
}
}
p.Rows++
touched[leaderOf[chunkKey{cs.Chapter, cs.ChunkIdx}]] = true
usd, fromHistory, moved := rp.usd(cs)
p.USD += usd
if fromHistory {
p.HistoricalRows++
}
if moved {
p.ModelMovedRows++
}
}
p.OutputUnits = len(touched)
return p, nil
}
// projectBookUSD extrapolates what the WHOLE book costs, from the units already FULLY attempted (done or
// flagged), at the CURRENT price table. It is the SINGLE definition of the number `tmctl status` reports
// as projected_book_usd — status used to compute it inline, and a threshold computed from a second,
// drifting definition of the same quantity is exactly the class of bug the memberDrops helper was
// extracted to remove.
//
// It re-prices for the same reason the re-payment amount does (row 181), and specifically so the consent
// gate cannot end up half-historical: the threshold is 5% of THIS, so leaving the base in old money
// while the amount moves to the new table would compare two different currencies and quietly change how
// often the operator is asked at all.
//
// It deliberately does NOT use book-committed spend: committed also carries partial spend on
// IN-PROGRESS units, which are outside the denominator and would over-estimate the book. Nor does it
// re-use resolveChunkState's CostUSD, which is the HISTORICAL sum the per-chapter passports report —
// what a chapter has cost is a fact, not a projection.
func projectBookUSD(units []editUnit, byChunk map[chunkKey][]store.ChunkStatus, nDraftStages, nEditStages int, rp *repricer) float64 {
var processedCost float64
processed := 0
for _, u := range units {
expected := len(u.Members)*nDraftStages + nEditStages
rows := unitRows(u, byChunk)
res := resolveChunkState(rows, expected)
if res.State != ChunkDone && res.State != ChunkFlagged {
continue
}
for _, cs := range rows {
usd, _, _ := rp.usd(cs)
processedCost += usd
}
processed++
}
if processed == 0 {
return 0
}
return processedCost / float64(processed) * float64(len(units))
}
// rebillConsentThreshold resolves the consent threshold for this book: the book's own
// `rebill_consent_usd` when it declares one, else the ratified `min($0.50, 5% × ProjectedBookUSD)` with
// the $0.50 floor for a book that has processed nothing yet.
func (r *Runner) rebillConsentThreshold(projectedBookUSD float64) (usd float64, source string) {
if r.Book.RebillConsentUSD > 0 {
return r.Book.RebillConsentUSD, "book.rebill_consent_usd"
}
if projectedBookUSD <= 0 {
return rebillConsentFloorUSD, "the $0.50 floor — the book has no processed units to take 5% of"
}
if share := rebillConsentShare * projectedBookUSD; share < rebillConsentFloorUSD {
return share, fmt.Sprintf("5%% of the projected book cost $%.6f", projectedBookUSD)
}
return rebillConsentFloorUSD, fmt.Sprintf("the $0.50 cap, under 5%% of the projected book cost $%.6f", projectedBookUSD)
}
// projectionBasis says, inside the consent text, what the amount is made of — and what it is NOT. It is
// the one sentence the operator's consent is given to, so it must not claim more than the number does.
//
// What it promises is the DIRECTION of the error, never accuracy (D39.150 п.1): the figure is the stored
// tokens at the model each stage resolves to today, floored at the answering model's price, and a model
// that has never seen these chunks may spend a different number of tokens. The two remainders the figure
// holds for are named rather than rounded away — units priced across a model move (an upper estimate) and
// units carried wholly or partly at last season's money (not re-priced at all, and possibly below today's
// table) — because an amount partly made of either is still a fact about the number. The same sentence
// serves `tmctl status` (StatusReport.RebillFigureBasis), so the two channels describe one figure in one
// wording.
func projectionBasis(p RebillProjection) string {
s := "their recorded tokens at the CURRENT price table, priced at the model each stage resolves to TODAY — an ESTIMATE that errs upward, never below what the answering model would charge for the same tokens where their usage is on file"
if p.ModelMovedRows > 0 {
s += fmt.Sprintf("; %d of the %d unit(s) were bought from a model their stage no longer resolves to and are priced at the dearer of the two where their usage is on file — a model that has never seen these chunks may spend a different number of tokens", p.ModelMovedRows, p.Rows)
}
if p.HistoricalRows > 0 {
s += fmt.Sprintf("; %d of the %d unit(s) carry, wholly or in part, the amount they were originally billed at — that much of their usage could not be re-priced and may sit below today's table", p.HistoricalRows, p.Rows)
}
return s
}
// checkRebillConsent is the gate: it refuses BEFORE any reservation when the run would re-pay more than
// the book's consent threshold and the operator has not consented to that amount.
//
// It is called from both write paths — TranslateBook (before the waves, hence before the first
// Reserve) and Redrive (before the DESTRUCTIVE reset, so a refusal cannot leave the flag telemetry
// deleted, the external-review 1c torn-state discipline). The $0 read-only surfaces (status / report /
// export, D20.4) never reach it, so a book that needs consent stays fully inspectable.
//
// It is deliberately NOT conditioned on r.Resnapshot. --resnapshot is the permission to RE-PIN, and a
// permission that names no amount cannot carry a Р6 consent to a concrete spend — that is precisely
// the debt this closes. It also covers the case --resnapshot does not: a run interrupted midway through
// a re-pin leaves jobs on the new snapshot while their chunk_status rows still carry the old one, and
// the next plain `translate` then re-bills them with no gate at all.
// `scope` is the run's VOLUME ceiling (volume.go), nil when none is in force. It produces TWO figures,
// and which of them each half of the gate is judged against is the whole correctness of this function.
//
// ⚠ THE THRESHOLD IS JUDGED AGAINST THE BOOK, NEVER AGAINST THE RUN — and the first version of this
// pack got that wrong. Scoping the amount to the admitted units and leaving the threshold book-wide
// sounds symmetrical and is not: with a volume ceiling the caller CHOOSES how small each run is, so
// `--resnapshot --max-units N` in a loop re-pays the entire book without a single run ever crossing the
// threshold and without consent being asked once. That is precisely the silent re-purchase Р6 exists to
// prevent, and "the change is inert without the flag" is true and beside the point: with the flag it
// opens the door the gate is the door for. A threshold a caller can defeat by splitting is not a
// threshold. So the question "must we ask at all" is answered by the book's whole drift, which does not
// shrink when a purchase does.
//
// ⚠ THE NAMED CAP IS JUDGED AGAINST THE RUN, because it is a different kind of statement. A threshold is
// the book's policy on when a human must be consulted; `--accept-rebill=X` is the caller's instruction
// "spend no more than X". Measuring an instruction about SPEND against work this run will not do would
// refuse a caller whose cap fully covers what they are about to be charged — the platform funds that cap
// from the run's own hold, so it is sized to the purchase, not to the book.
//
// With no ceiling in force the two projections are the same object and exactly one is computed, so every
// existing run is judged by byte-identical figures.
func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk, scope *volumeScope) error {
statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
if err != nil {
return fmt.Errorf("pipeline: read chunk_status for the re-bill projection: %w", err)
}
rp, err := r.newRepricer()
if err != nil {
return err
}
// A write path already holds the real split, so the lazy provider just hands it back — no second
// ingest, and no branch where the consent gate could see text-free chunks.
withText := func() ([]chunk.Chunk, error) { return chunks, nil }
book, err := r.projectRebill(statuses, chunks, withText, rp)
if err != nil {
return err
}
if book.Rows == 0 {
return nil // nothing already-paid is superseded — this run bills only new work
}
// What THIS run will actually re-pay. Only computed when a ceiling narrows it; otherwise it IS the
// book's figure, which keeps the un-bounded path at one projection exactly as before.
proj := book
if scope != nil {
if proj, err = r.projectRebill(scope.admittedStatuses(statuses), chunks, withText, rp); err != nil {
return err
}
}
if proj.Rows == 0 {
// The book carries drift, but none of it is inside what this run was granted, so this run re-pays
// nothing and there is no concrete spend to consent to. Not a hole in the threshold above: the
// moment a run is granted a superseded unit, proj stops being empty and the book-wide threshold
// below decides — so splitting cannot walk past the gate, it can only postpone meeting it.
return nil
}
// The denominator is the WHOLE book's stored rows, unfiltered by the volume scope: the threshold is
// "5% of what this book costs", a property of the book.
byChunk := map[chunkKey][]store.ChunkStatus{}
for _, cs := range statuses {
byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs)
}
bookUSD := projectBookUSD(r.outputUnits(chunks), byChunk,
len(r.waveStagesIndexed(waveDraft)), len(r.waveStagesIndexed(waveEdit)), rp)
// ⛔ PLUS THE MONEY OF THE BANK ROLES, which this projection did not see at all (backlog row 194).
//
// The terminologist and the classifier are checkpointed under a SYNTHETIC stage at chapter 0 and write
// no chunk_status row — deliberately, they are not a wave — so every derivation that walks chunk_status
// is structurally blind to them. Measured on the cold run: $0.04980482 of $0.43610966, 11.4% of the
// run, and not one cent of it in the number an operator is shown before deciding to buy more.
//
// ⚠ WHAT THIS ADDEND IS, EXACTLY, because the honest bound matters more than the figure: it is the
// contour ALREADY COMMITTED, not a forecast of it. The contour does not scale with units — it is a
// per-BOOK pass whose batches are re-formed whenever the drafted set grows (row 233) — so extrapolating
// it per unit would invent a number. Adding what has already been spent makes the projection a LOWER
// bound on the truth instead of an omission of a whole class, and «lower bound» is what this ledger has
// always been (row 78).
//
// It moves NO figure the platform reads: committed_usd is summed from `spend`, which already holds this
// money — the ledger was never blind, the PROJECTION was.
// ⛔ THE CONSENT THRESHOLD IS COMPUTED WITHOUT THE CONTOUR, and the two numbers part company HERE.
//
// The threshold governs RE-PAYMENT: it is 5% of what the book costs, and it decides when an operator
// must be asked before already-billed work is bought again. The bank-role contour is not re-paid by a
// snapshot move — it is re-bought when the DRAFTED SET grows (backlog row 233), on its own axis — so
// folding it into the base would raise the bar for asking without adding anything the bar is about,
// i.e. make a money gate quietly WEAKER. The first version of this fix did exactly that by adding the
// contour before the threshold was taken; caught by acceptance, and the split is the orchestrator's
// decision (31.08), not this file's.
//
// The REPORTED projection still carries the contour and is computed where it is published
// (status.go's ProjectedBookUSD): «what will this book cost me» must not omit a whole class of spend.
// Same money, two questions, and only one of them is about re-payment — so the addend belongs to the
// answer that is about the book's cost, and to that one only.
threshold, source := r.rebillConsentThreshold(bookUSD)
// A NAMED ceiling is an instruction, not merely a consent form: it is honoured even below the
// threshold, so an operator who wrote "no more than $X" is never billed $X+ε on the grounds that the
// amount was small enough not to need asking.
if r.AcceptRebill.Capped && proj.USD > r.AcceptRebill.CapUSD {
// The figure travels with its basis here exactly as it does in the threshold refusal below (D39.181
// п.1): a caller who named a cap is deciding whether to raise it, and a bare number cannot tell them
// how much of it is an upper estimate or last season's money. The basis is THIS run's, because the
// figure is — every number in a sentence comes from the projection the sentence is about.
return fmt.Errorf("pipeline: the projected re-payment is ~$%.6f (%d chunk×stage unit(s) already billed under a superseded snapshot; %s) but --accept-rebill=%g caps consent at $%.6f — refusing. NOTHING was reserved and no row was touched. Raise the ceiling, or pass a bare --accept-rebill to accept the full projected amount",
proj.USD, proj.Rows, projectionBasis(proj), r.AcceptRebill.CapUSD, r.AcceptRebill.CapUSD)
}
if r.AcceptRebill.Given {
r.Log.WarnContext(ctx, "accepting a projected re-payment of already-billed work (--accept-rebill)",
"rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD),
"book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD),
"units_not_repriced", proj.HistoricalRows, "units_priced_across_a_model_move", proj.ModelMovedRows,
"repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold))
return nil
}
// ⚠ THE BOOK'S figure is what the threshold judges, not this run's. See the doc comment: a caller
// that chooses how small each run is could otherwise re-pay the whole book a slice at a time and
// never once be asked. `book` equals `proj` whenever no volume ceiling is in force, so this is the
// same comparison it has always been for every un-bounded run.
if book.USD <= threshold {
// Under the threshold the run continues without friction (the ratified behaviour: a term append
// touching three chunks costs cents). Continuing is not the same as being silent — the amount is
// money and it goes to the log.
r.Log.InfoContext(ctx, "re-paying already-billed work under the consent threshold; continuing without asking",
"rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD),
"book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD),
"units_not_repriced", proj.HistoricalRows, "units_priced_across_a_model_move", proj.ModelMovedRows,
"repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold))
return nil
}
hint := ""
if !r.Resnapshot {
hint = " The run also needs --resnapshot: without it the superseded jobs stop it anyway."
}
// ⚠ EVERY FIGURE IN A SENTENCE MUST COME FROM THE SAME PROJECTION AS THE SENTENCE. This parenthetical
// hangs off the BOOK-wide clause, so it takes the BOOK's re-pin count. The first version spliced the
// volume-scoped one in, which under an active ceiling printed a smaller number inside a sentence whose
// every other figure was book-wide — an operator adding them up would have got a book that did not
// exist. Two independent review lenses caught the same splice, which is what mixed-provenance numbers
// in one sentence reliably produce.
repin := ""
if book.Repinned > 0 {
repin = fmt.Sprintf(" (%d further unit(s) of the book are re-pinned for $0 — the bank moved but their injected bytes did not)", book.Repinned)
}
// The two projections are printed SEPARATELY when a volume ceiling makes them differ, because
// collapsing them is how this gate goes wrong in either direction: quoting only the book's total at a
// small purchase asks for consent to money nobody will be charged, and quoting only the run's slice
// hides that the book is being re-bought a slice at a time. The scoped clause carries the scoped
// re-pin count for the same provenance reason.
scoped := ""
if book.Rows != proj.Rows || book.USD != proj.USD {
scopedRepin := ""
if proj.Repinned != book.Repinned {
scopedRepin = fmt.Sprintf(" (and re-pin %d of them for $0)", proj.Repinned)
}
// The scoped figure travels with the SCOPED projection's basis, as every figure does with its own:
// which of the run's units are an upper estimate or old money is not a fact the book's basis can
// state for them.
scoped = fmt.Sprintf(" THIS run, bounded by --max-units, would re-pay %d of them, ~$%.6f (%s)%s — but the threshold is judged against the book, because a ceiling the caller sizes could otherwise re-pay the whole book a slice at a time without ever being asked.",
proj.Rows, proj.USD, projectionBasis(proj), scopedRepin)
}
return fmt.Errorf("pipeline: this run would RE-PAY for work already billed: %d chunk×stage unit(s) of this book are resolved under a superseded snapshot and would be paid for again, ~$%.6f (%s)%s.%s That is over this book's consent threshold $%.6f (%s), and Р6 requires consent to a CONCRETE spend, not a blanket one (D20.2-Q2). NOTHING was reserved and no row was touched. Re-run with --accept-rebill to accept the whole projected amount, or --accept-rebill=<usd> to accept it only up to a ceiling (the ceiling is measured against what THIS run re-pays, and one below that refuses).%s",
book.Rows, book.USD, projectionBasis(book), repin, scoped, threshold, source, hint)
}
// sourceMovedUnderTheRows decides whether the re-payment projection has to reproduce the rendered content
// hashes at all.
//
// The question it answers is «could any rendered byte have moved since the stored rows were written», and
// the stored manifest answers it: its validity key folds the source SHA, the encoding, the chunker
// version, the segmentation budget, the language pack and the embedded data — every input the rendered
// text is a function of. A manifest that still validates proves the source under those rows is the source
// they were rendered from, and a same-snapshot row really does resume for $0.
//
// ⛔ IT IS ANSWERED ONCE AND CACHED, AND THAT IS CORRECTNESS, NOT SPEED. `translate` PERSISTS the manifest
// before the consent gate runs (bookrun.go — the sidecar is written where the split every paid byte is
// addressed against was just computed). So an un-cached probe, asked after that point, reads a sidecar
// that already describes the NEW source, answers «nothing moved», and the expensive check is skipped on
// the one surface where money is actually authorised: the re-payment consent gate would not fire on an
// in-place source edit at all. The first version of this fix had exactly that hole — right on the read
// path, dead on the money path — and the acceptance found it. `noteSourceVintage` is what freezes the
// answer while the sidecar still describes the rows.
//
// Caching also removes the cost the un-cached form had: it was asked PER ROW, and each ask re-read the
// sidecar and re-hashed the whole source — O(rows × bytes) inside a $0 read command.
//
// The direction of every failure is «assume it moved»: no manifest, an unreadable one, one written in
// another document version, one whose key no longer matches — all mean the projection must do the
// expensive check rather than promise $0. That is the safe direction for money: over-counting a
// re-payment makes an operator consent to more than will be spent, under-counting charges him for work he
// was told was free.
func (r *Runner) sourceMovedUnderTheRows() bool {
if r.rowsSourceMoved == nil {
moved := r.loadManifest() == nil
r.rowsSourceMoved = &moved
}
return *r.rowsSourceMoved
}
// noteSourceVintage freezes the answer above while the stored manifest still describes the SOURCE THE
// STORED ROWS WERE WRITTEN UNDER. It must be called before anything rewrites that sidecar.
func (r *Runner) noteSourceVintage() { _ = r.sourceMovedUnderTheRows() }
// bankRoleCommittedUSD is what the book has already paid for its bank roles — the money that lives in
// checkpoints under the synthetic terminology stage and in no chunk_status row.
//
// A read failure degrades to zero and says so: the projection is a decision aid, and it must not be able
// to stop a $0 read. Silence would be the defect, so the WARN says what the number is missing.
func (r *Runner) bankRoleCommittedUSD() float64 {
var total float64
for _, role := range []string{roleTerminologist, roleClassifier} {
usd, err := r.Store.RoleSpentUSD(r.Book.BookID, role)
if err != nil {
r.Log.Warn("the bank-role spend could not be read; the book projection is missing that whole class of money, and is therefore a LOWER bound with an unknown gap rather than with a named one",
"book", r.Book.BookID, "role", role, "err", err)
continue
}
total += usd
}
return total
}