package pipeline import ( "context" "fmt" "textmachine/backend/internal/chunk" "textmachine/backend/internal/store" ) // rebill.go: CONSENT TO A RE-PAYMENT (D20.2-Q2, spec §7.3 of backend/docs/D15.2-*.md) — the Р6 // "consent to a CONCRETE spend" gate standing in front of every path that would bill again for work // this book has already been billed for. // // GRANULARITY (ratified with this pack). The spec's §7 projection is PER-CHUNK and rides `guard_hash`, // which does not exist in this engine at all: drift here is BOOLEAN (a stored row's snapshot either is // or is not the current one) and a snapshot move misses EVERY checkpoint of its wave. So consent is // built at the granularity the engine actually re-pays at — the SNAPSHOT — and the projection is the // honest reading of that: "the units already paid for under a superseded snapshot will be paid for // again; their recorded tokens at TODAY'S price table come to $X" (reprice.go, row 181 — until then it // summed the historical cost_usd, which is a different table the moment a vendor moves its prices). // // Two consequences of that granularity, both deliberate: // - the spec's §7.1-бис editor-cascade fix ("a stage strictly below a re-billed unit is itself // re-billed") is DEGENERATE here — everything of the moved wave is re-billed already, so the // cascade rule adds nothing. Its ratified ordering ("the cascade lands BEFORE the flag semantics") // is therefore not violated but inapplicable; // - the spec's EstimateUSD fallback ("for cascade units with no past price") has no subject either: // every unit counted here HAS stored usage, because it is counted precisely for having been paid. // A paid row that re-prices to $0 is a genuinely $0 model (local/priced-zero), so 0 is its honest // contribution rather than a gap to estimate around. // // WHAT IS NOT PROJECTED (documented, not silent): the CONTENT axis. A source edit that leaves the chunk // manifest intact moves a chunk's content_hash but not the snapshot, so it re-bills exactly the touched // chunks and is invisible here — the same caveat status.bookChunks carries, and the half that belongs // to content-addressed resume (v3), not to this gate. // Ratified default threshold (D20.2-Q2): `min($0.50, 5% × ProjectedBookUSD)`, with the $0.50 acting as // an absolute FLOOR when the book has no processed units yet (the 5% branch would then be $0 and would // demand consent for a one-cent append). A book may override it with `rebill_consent_usd`. const ( rebillConsentFloorUSD = 0.50 rebillConsentShare = 0.05 ) // RebillConsent is the operator's answer to `--accept-rebill[=usd]`: the Р6 consent to a CONCRETE // spend. A bare flag accepts whatever the projection turns out to be; `--accept-rebill=1.50` accepts it // only while it stays at or below $1.50, so the consent names an amount rather than a blanket. type RebillConsent struct { Given bool // the flag was passed at all Capped bool // a ceiling was named (--accept-rebill=) CapUSD float64 // that ceiling; meaningful only when Capped } // RebillProjection is what a run would re-pay: the chunk×stage units resolved under a superseded // snapshot, and what buying them again would cost AT THE CURRENT PRICE TABLE (reprice.go). type RebillProjection struct { Rows int USD float64 // HistoricalRows counts the units inside USD that could not be fully re-priced and carry the amount // they were billed at instead: a billed-decode row with no usage on file, a row whose calls no longer // account for its money, or one whose newest calls are provably not its own (reprice.go). It exists so // the operator-facing text can name what the number is made of rather than claim a re-pricing it did // not achieve. HistoricalRows int // ModelMovedRows counts the units inside USD whose stage resolves today to a model other than the one // their calls were SENT to (reprice.go — the comparison is against the requested slug, so a provider // canonicalising its own name is not a move): they are priced at the dearer of the two, and the token // count the new model would spend is unknowable — so the larger this count, the more USD is an upper // ESTIMATE rather than a re-pricing of known tokens. It is the number the consent text names when it // says so (D39.150 п.1: the caveat travels with the figure). ModelMovedRows int // OutputUnits is the same re-payment counted in OUTPUT UNITS — the granularity the manifest publishes // as units_total, that `--max-units` bounds, and that the platform sells chapters in. // // ⚠ IT EXISTS BECAUSE `Rows` IS A DIFFERENT UNIT WEARING THE SAME WORD. Rows counts chunk×stage, the // BILLING unit; every other "units" number the engine puts on the wire (total_units, progress totals, // chapters[].units_total) is the OUTPUT unit. Measured on a three-chapter fixture, one document carried // `rebill_units: 15` two lines under `total_units: 6` with nothing marking the difference — and the // ratio is not a constant a consumer could divide out: it is len(Members)·nDraftStages + nEditStages, // which varies unit by unit WITHIN one book and whose factors never cross the seam at all. A platform // sizing a re-pass from the estimate and passing that number to --max-units would buy several times the // book it meant to, which is this pack's own defect reproduced one layer down. OutputUnits int // Repinned counts the units the run will serve for $0 despite a moved snapshot (pack-20 point 5): the // move was bank-only and their rendered bytes are unchanged. It is not part of the amount — it is the // number that makes the amount believable, because before pack-20 every one of these was counted as a // re-payment and the operator was asked to consent to a whole wave for one signed word. Repinned int } // projectRebill sums the units that a run under the CURRENT config would pay for a second time. // // A unit counts when all three hold: it carries a snapshot (a row written by a real run), it was BILLED // (ok/flagged — a `skipped` row never reached a provider and cost nothing), and its snapshot differs // from what its own WAVE renders now. The per-wave comparison is load-bearing: a bank-mining enrichment // moves only the edit-wave snapshot, and projecting the draft wave against it would report the whole // book as re-billed when the drafts in fact resume at $0 (the "re-paid ONCE" invariant). // // Two kinds of stored row are ORPHANS — they exist but nothing will run them again, so counting them // would ask the operator to consent to money that will not be spent: // - a row whose STAGE the current pipeline no longer has (a renamed/retired stage); // - a row whose CHUNK the current manifest no longer has (the source was shortened, so the position // is gone). Both waves address their rows at manifest chunk positions — a draft row at its own // chunk, an edit row at its unit's leader chunk — so one membership test covers both. // // Over-estimating is the safe direction for a consent gate, but a number the operator is asked to // approve and then not charged is exactly what makes such a number stop being read. Which is why the // amount is not a safety margin either way: it is the stored tokens re-priced through the same seam the // reservation will use, so the two answer the same question with the same table. // // CONTENT AWARENESS (pack-20 point 5, closing the phase-1 P3 finding). Snapshot divergence alone // over-counts: since the bank is folded per wave, signing ONE term marked the whole edit wave as re-billed // even though almost every unit's rendered bytes are unchanged. The projection now models the resume // predicate the run actually applies (repin.go): a bank-only snapshot move whose unit renders the SAME // content hash resumes at $0 and is counted as re-pinned, not re-paid. What remains in the amount is the // units where the changed term genuinely occurs — the "N units, ~$X" the owner asked to see before // consenting. The reproduction is $0 (string rendering + store reads, no provider), and a position whose // inputs cannot be reproduced falls back to the conservative answer: counted as a re-payment. // // The wave snapshots are rendered LAZILY, so a draft-only pipeline never renders an edit-wave snapshot // it has no stages for. // // `manifest` supplies only the POSITIONS still in the book (the membership test above), which is why it // may be the text-free projection of the persisted manifest (manifest.go). `withText` supplies the same // chunks WITH their source, and is called only on the content-aware branch below — the one that has to // re-render injected bytes and therefore genuinely needs the text. Splitting the two is what lets a // status read skip a 1.4 s re-chunk it almost never needs, without ever computing a content hash over an // empty string. func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Chunk, withText func() ([]chunk.Chunk, error), rp *repricer) (RebillProjection, error) { var p RebillProjection decider := newRepinDecider(r) var contentHashes map[chunkKey]map[string]string live := make(map[chunkKey]bool, len(manifest)) for _, ch := range manifest { live[chunkKey{ch.Chapter, ch.ChunkIdx}] = true } // Which OUTPUT unit each position belongs to, so the same re-payment can be reported in the unit the // seam sells in as well as the one it bills in. An edit row lives at its unit's leader chunk and a // leader is one of the unit's members, so this one map covers both waves. leaderOf := make(map[chunkKey]chunkKey, len(manifest)) for _, u := range r.outputUnits(manifest) { leader := chunkKey{u.Chapter, u.FirstChunkIdx} for _, m := range u.Members { leaderOf[chunkKey{m.Chapter, m.ChunkIdx}] = leader } } // ONE lazy reproduction of the rendered content hashes, shared by both branches that need them. It was // inline in the bank-only branch; the source-edit check needs the same map, and a second construction // site is how the two would come to disagree about what «the bytes this run would render» means. reproduce := func() (map[chunkKey]map[string]string, error) { if contentHashes != nil { return contentHashes, nil } full, ferr := withText() if ferr != nil { return nil, fmt.Errorf("pipeline: re-chunk the source for the re-bill content check: %w", ferr) } contentHashes = r.cachedRenderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)) return contentHashes, nil } touched := map[chunkKey]bool{} draftNames := stageNameSet(r.waveStagesIndexed(waveDraft)) editNames := stageNameSet(r.waveStagesIndexed(waveEdit)) rendered := map[wave]string{} current := func(w wave) (string, error) { if s, ok := rendered[w]; ok { return s, nil } s, _, err := r.snapshotIDForWave(w) if err != nil { return "", fmt.Errorf("pipeline: render the current snapshot for the re-bill projection: %w", err) } rendered[w] = s return s, nil } for _, cs := range statuses { if cs.SnapshotID == "" || cs.Disposition == string(DispSkipped) { continue } if !live[chunkKey{cs.Chapter, cs.ChunkIdx}] { continue // the position is gone from the manifest — nothing will re-run it } var w wave switch { case draftNames[cs.Stage]: w = waveDraft case editNames[cs.Stage]: w = waveEdit default: continue // a stage the current pipeline does not run is never re-billed } cur, err := current(w) if err != nil { return p, err } if cs.SnapshotID == cur { // ⛔ THE SNAPSHOT MATCHING IS NOT ENOUGH, and this branch used to stop here (backlog row 238). // The source is deliberately NOT in the snapshot, so an edit to the source file moves the // row's rendered CONTENT while its snapshot id stays identical — and the run's own resume // fast-path checks exactly that (`cs.ContentHash == contentHash`, stagerun.go) and re-buys the // unit. The projection said «$0 resume» and `translate` charged: money silent about work it // was about to do (disclosure law §2.1). The docstring above claimed this function «models the // resume predicate the run actually applies» — it modelled it in the bank-only branch only, // which made the claim itself an instance of §2.2. // // The check is CONDITIONAL on a cheap probe rather than always-on, and that is deliberate: the // content hashes cost a re-chunk of the source (~1.4 s on a 23 MB book) that `status` almost // never needs, and the split exists to let a status read skip it. sourceMovedUnderTheRows // answers «could any rendered byte have moved» from the stored manifest's validity key, which // folds the source SHA — so the expensive answer is only bought once there is a question. if !r.sourceMovedUnderTheRows() { continue // resumes at $0, and the probe says no input under it moved } hashes, herr := reproduce() if herr != nil { return p, herr } if want, ok := hashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && want == cs.ContentHash { continue // the source moved somewhere, but not under THIS row: still a $0 resume } // Either the rendered bytes differ or they cannot be reproduced. Both fall through to the // conservative answer — the same direction the rest of this function takes — and the row is // counted as a re-payment, which is what the run will actually do. } // The snapshot moved — but that is not yet a re-payment. Ask the two questions the resume path asks. bankOnly, berr := decider.bankOnlyMove(cs.SnapshotID, w) if berr != nil { return p, berr } if bankOnly { if contentHashes == nil { // rendered once, lazily: a book with no bank move never pays for it if _, ferr := reproduce(); ferr != nil { return p, ferr } } if h, ok := contentHashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && h == cs.ContentHash { p.Repinned++ continue // the bank moved somewhere else in the book; this unit is re-pinned for $0 } } p.Rows++ touched[leaderOf[chunkKey{cs.Chapter, cs.ChunkIdx}]] = true usd, fromHistory, moved := rp.usd(cs) p.USD += usd if fromHistory { p.HistoricalRows++ } if moved { p.ModelMovedRows++ } } p.OutputUnits = len(touched) return p, nil } // projectBookUSD extrapolates what the WHOLE book costs, from the units already FULLY attempted (done or // flagged), at the CURRENT price table. It is the SINGLE definition of the number `tmctl status` reports // as projected_book_usd — status used to compute it inline, and a threshold computed from a second, // drifting definition of the same quantity is exactly the class of bug the memberDrops helper was // extracted to remove. // // It re-prices for the same reason the re-payment amount does (row 181), and specifically so the consent // gate cannot end up half-historical: the threshold is 5% of THIS, so leaving the base in old money // while the amount moves to the new table would compare two different currencies and quietly change how // often the operator is asked at all. // // It deliberately does NOT use book-committed spend: committed also carries partial spend on // IN-PROGRESS units, which are outside the denominator and would over-estimate the book. Nor does it // re-use resolveChunkState's CostUSD, which is the HISTORICAL sum the per-chapter passports report — // what a chapter has cost is a fact, not a projection. func projectBookUSD(units []editUnit, byChunk map[chunkKey][]store.ChunkStatus, nDraftStages, nEditStages int, rp *repricer) float64 { var processedCost float64 processed := 0 for _, u := range units { expected := len(u.Members)*nDraftStages + nEditStages rows := unitRows(u, byChunk) res := resolveChunkState(rows, expected) if res.State != ChunkDone && res.State != ChunkFlagged { continue } for _, cs := range rows { usd, _, _ := rp.usd(cs) processedCost += usd } processed++ } if processed == 0 { return 0 } return processedCost / float64(processed) * float64(len(units)) } // rebillConsentThreshold resolves the consent threshold for this book: the book's own // `rebill_consent_usd` when it declares one, else the ratified `min($0.50, 5% × ProjectedBookUSD)` with // the $0.50 floor for a book that has processed nothing yet. func (r *Runner) rebillConsentThreshold(projectedBookUSD float64) (usd float64, source string) { if r.Book.RebillConsentUSD > 0 { return r.Book.RebillConsentUSD, "book.rebill_consent_usd" } if projectedBookUSD <= 0 { return rebillConsentFloorUSD, "the $0.50 floor — the book has no processed units to take 5% of" } if share := rebillConsentShare * projectedBookUSD; share < rebillConsentFloorUSD { return share, fmt.Sprintf("5%% of the projected book cost $%.6f", projectedBookUSD) } return rebillConsentFloorUSD, fmt.Sprintf("the $0.50 cap, under 5%% of the projected book cost $%.6f", projectedBookUSD) } // projectionBasis says, inside the consent text, what the amount is made of — and what it is NOT. It is // the one sentence the operator's consent is given to, so it must not claim more than the number does. // // What it promises is the DIRECTION of the error, never accuracy (D39.150 п.1): the figure is the stored // tokens at the model each stage resolves to today, floored at the answering model's price, and a model // that has never seen these chunks may spend a different number of tokens. The two remainders the figure // holds for are named rather than rounded away — units priced across a model move (an upper estimate) and // units carried wholly or partly at last season's money (not re-priced at all, and possibly below today's // table) — because an amount partly made of either is still a fact about the number. The same sentence // serves `tmctl status` (StatusReport.RebillFigureBasis), so the two channels describe one figure in one // wording. func projectionBasis(p RebillProjection) string { s := "their recorded tokens at the CURRENT price table, priced at the model each stage resolves to TODAY — an ESTIMATE that errs upward, never below what the answering model would charge for the same tokens where their usage is on file" if p.ModelMovedRows > 0 { s += fmt.Sprintf("; %d of the %d unit(s) were bought from a model their stage no longer resolves to and are priced at the dearer of the two where their usage is on file — a model that has never seen these chunks may spend a different number of tokens", p.ModelMovedRows, p.Rows) } if p.HistoricalRows > 0 { s += fmt.Sprintf("; %d of the %d unit(s) carry, wholly or in part, the amount they were originally billed at — that much of their usage could not be re-priced and may sit below today's table", p.HistoricalRows, p.Rows) } return s } // checkRebillConsent is the gate: it refuses BEFORE any reservation when the run would re-pay more than // the book's consent threshold and the operator has not consented to that amount. // // It is called from both write paths — TranslateBook (before the waves, hence before the first // Reserve) and Redrive (before the DESTRUCTIVE reset, so a refusal cannot leave the flag telemetry // deleted, the external-review 1c torn-state discipline). The $0 read-only surfaces (status / report / // export, D20.4) never reach it, so a book that needs consent stays fully inspectable. // // It is deliberately NOT conditioned on r.Resnapshot. --resnapshot is the permission to RE-PIN, and a // permission that names no amount cannot carry a Р6 consent to a concrete spend — that is precisely // the debt this closes. It also covers the case --resnapshot does not: a run interrupted midway through // a re-pin leaves jobs on the new snapshot while their chunk_status rows still carry the old one, and // the next plain `translate` then re-bills them with no gate at all. // `scope` is the run's VOLUME ceiling (volume.go), nil when none is in force. It produces TWO figures, // and which of them each half of the gate is judged against is the whole correctness of this function. // // ⚠ THE THRESHOLD IS JUDGED AGAINST THE BOOK, NEVER AGAINST THE RUN — and the first version of this // pack got that wrong. Scoping the amount to the admitted units and leaving the threshold book-wide // sounds symmetrical and is not: with a volume ceiling the caller CHOOSES how small each run is, so // `--resnapshot --max-units N` in a loop re-pays the entire book without a single run ever crossing the // threshold and without consent being asked once. That is precisely the silent re-purchase Р6 exists to // prevent, and "the change is inert without the flag" is true and beside the point: with the flag it // opens the door the gate is the door for. A threshold a caller can defeat by splitting is not a // threshold. So the question "must we ask at all" is answered by the book's whole drift, which does not // shrink when a purchase does. // // ⚠ THE NAMED CAP IS JUDGED AGAINST THE RUN, because it is a different kind of statement. A threshold is // the book's policy on when a human must be consulted; `--accept-rebill=X` is the caller's instruction // "spend no more than X". Measuring an instruction about SPEND against work this run will not do would // refuse a caller whose cap fully covers what they are about to be charged — the platform funds that cap // from the run's own hold, so it is sized to the purchase, not to the book. // // With no ceiling in force the two projections are the same object and exactly one is computed, so every // existing run is judged by byte-identical figures. func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk, scope *volumeScope) error { statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID) if err != nil { return fmt.Errorf("pipeline: read chunk_status for the re-bill projection: %w", err) } rp, err := r.newRepricer() if err != nil { return err } // A write path already holds the real split, so the lazy provider just hands it back — no second // ingest, and no branch where the consent gate could see text-free chunks. withText := func() ([]chunk.Chunk, error) { return chunks, nil } book, err := r.projectRebill(statuses, chunks, withText, rp) if err != nil { return err } if book.Rows == 0 { return nil // nothing already-paid is superseded — this run bills only new work } // What THIS run will actually re-pay. Only computed when a ceiling narrows it; otherwise it IS the // book's figure, which keeps the un-bounded path at one projection exactly as before. proj := book if scope != nil { if proj, err = r.projectRebill(scope.admittedStatuses(statuses), chunks, withText, rp); err != nil { return err } } if proj.Rows == 0 { // The book carries drift, but none of it is inside what this run was granted, so this run re-pays // nothing and there is no concrete spend to consent to. Not a hole in the threshold above: the // moment a run is granted a superseded unit, proj stops being empty and the book-wide threshold // below decides — so splitting cannot walk past the gate, it can only postpone meeting it. return nil } // The denominator is the WHOLE book's stored rows, unfiltered by the volume scope: the threshold is // "5% of what this book costs", a property of the book. byChunk := map[chunkKey][]store.ChunkStatus{} for _, cs := range statuses { byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs) } bookUSD := projectBookUSD(r.outputUnits(chunks), byChunk, len(r.waveStagesIndexed(waveDraft)), len(r.waveStagesIndexed(waveEdit)), rp) // ⛔ PLUS THE MONEY OF THE BANK ROLES, which this projection did not see at all (backlog row 194). // // The terminologist and the classifier are checkpointed under a SYNTHETIC stage at chapter 0 and write // no chunk_status row — deliberately, they are not a wave — so every derivation that walks chunk_status // is structurally blind to them. Measured on the cold run: $0.04980482 of $0.43610966, 11.4% of the // run, and not one cent of it in the number an operator is shown before deciding to buy more. // // ⚠ WHAT THIS ADDEND IS, EXACTLY, because the honest bound matters more than the figure: it is the // contour ALREADY COMMITTED, not a forecast of it. The contour does not scale with units — it is a // per-BOOK pass whose batches are re-formed whenever the drafted set grows (row 233) — so extrapolating // it per unit would invent a number. Adding what has already been spent makes the projection a LOWER // bound on the truth instead of an omission of a whole class, and «lower bound» is what this ledger has // always been (row 78). // // It moves NO figure the platform reads: committed_usd is summed from `spend`, which already holds this // money — the ledger was never blind, the PROJECTION was. // ⛔ THE CONSENT THRESHOLD IS COMPUTED WITHOUT THE CONTOUR, and the two numbers part company HERE. // // The threshold governs RE-PAYMENT: it is 5% of what the book costs, and it decides when an operator // must be asked before already-billed work is bought again. The bank-role contour is not re-paid by a // snapshot move — it is re-bought when the DRAFTED SET grows (backlog row 233), on its own axis — so // folding it into the base would raise the bar for asking without adding anything the bar is about, // i.e. make a money gate quietly WEAKER. The first version of this fix did exactly that by adding the // contour before the threshold was taken; caught by acceptance, and the split is the orchestrator's // decision (31.08), not this file's. // // The REPORTED projection still carries the contour and is computed where it is published // (status.go's ProjectedBookUSD): «what will this book cost me» must not omit a whole class of spend. // Same money, two questions, and only one of them is about re-payment — so the addend belongs to the // answer that is about the book's cost, and to that one only. threshold, source := r.rebillConsentThreshold(bookUSD) // A NAMED ceiling is an instruction, not merely a consent form: it is honoured even below the // threshold, so an operator who wrote "no more than $X" is never billed $X+ε on the grounds that the // amount was small enough not to need asking. if r.AcceptRebill.Capped && proj.USD > r.AcceptRebill.CapUSD { // The figure travels with its basis here exactly as it does in the threshold refusal below (D39.181 // п.1): a caller who named a cap is deciding whether to raise it, and a bare number cannot tell them // how much of it is an upper estimate or last season's money. The basis is THIS run's, because the // figure is — every number in a sentence comes from the projection the sentence is about. return fmt.Errorf("pipeline: the projected re-payment is ~$%.6f (%d chunk×stage unit(s) already billed under a superseded snapshot; %s) but --accept-rebill=%g caps consent at $%.6f — refusing. NOTHING was reserved and no row was touched. Raise the ceiling, or pass a bare --accept-rebill to accept the full projected amount", proj.USD, proj.Rows, projectionBasis(proj), r.AcceptRebill.CapUSD, r.AcceptRebill.CapUSD) } if r.AcceptRebill.Given { r.Log.WarnContext(ctx, "accepting a projected re-payment of already-billed work (--accept-rebill)", "rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD), "book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD), "units_not_repriced", proj.HistoricalRows, "units_priced_across_a_model_move", proj.ModelMovedRows, "repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold)) return nil } // ⚠ THE BOOK'S figure is what the threshold judges, not this run's. See the doc comment: a caller // that chooses how small each run is could otherwise re-pay the whole book a slice at a time and // never once be asked. `book` equals `proj` whenever no volume ceiling is in force, so this is the // same comparison it has always been for every un-bounded run. if book.USD <= threshold { // Under the threshold the run continues without friction (the ratified behaviour: a term append // touching three chunks costs cents). Continuing is not the same as being silent — the amount is // money and it goes to the log. r.Log.InfoContext(ctx, "re-paying already-billed work under the consent threshold; continuing without asking", "rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD), "book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD), "units_not_repriced", proj.HistoricalRows, "units_priced_across_a_model_move", proj.ModelMovedRows, "repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold)) return nil } hint := "" if !r.Resnapshot { hint = " The run also needs --resnapshot: without it the superseded jobs stop it anyway." } // ⚠ EVERY FIGURE IN A SENTENCE MUST COME FROM THE SAME PROJECTION AS THE SENTENCE. This parenthetical // hangs off the BOOK-wide clause, so it takes the BOOK's re-pin count. The first version spliced the // volume-scoped one in, which under an active ceiling printed a smaller number inside a sentence whose // every other figure was book-wide — an operator adding them up would have got a book that did not // exist. Two independent review lenses caught the same splice, which is what mixed-provenance numbers // in one sentence reliably produce. repin := "" if book.Repinned > 0 { repin = fmt.Sprintf(" (%d further unit(s) of the book are re-pinned for $0 — the bank moved but their injected bytes did not)", book.Repinned) } // The two projections are printed SEPARATELY when a volume ceiling makes them differ, because // collapsing them is how this gate goes wrong in either direction: quoting only the book's total at a // small purchase asks for consent to money nobody will be charged, and quoting only the run's slice // hides that the book is being re-bought a slice at a time. The scoped clause carries the scoped // re-pin count for the same provenance reason. scoped := "" if book.Rows != proj.Rows || book.USD != proj.USD { scopedRepin := "" if proj.Repinned != book.Repinned { scopedRepin = fmt.Sprintf(" (and re-pin %d of them for $0)", proj.Repinned) } // The scoped figure travels with the SCOPED projection's basis, as every figure does with its own: // which of the run's units are an upper estimate or old money is not a fact the book's basis can // state for them. scoped = fmt.Sprintf(" THIS run, bounded by --max-units, would re-pay %d of them, ~$%.6f (%s)%s — but the threshold is judged against the book, because a ceiling the caller sizes could otherwise re-pay the whole book a slice at a time without ever being asked.", proj.Rows, proj.USD, projectionBasis(proj), scopedRepin) } return fmt.Errorf("pipeline: this run would RE-PAY for work already billed: %d chunk×stage unit(s) of this book are resolved under a superseded snapshot and would be paid for again, ~$%.6f (%s)%s.%s That is over this book's consent threshold $%.6f (%s), and Р6 requires consent to a CONCRETE spend, not a blanket one (D20.2-Q2). NOTHING was reserved and no row was touched. Re-run with --accept-rebill to accept the whole projected amount, or --accept-rebill= to accept it only up to a ceiling (the ceiling is measured against what THIS run re-pays, and one below that refuses).%s", book.Rows, book.USD, projectionBasis(book), repin, scoped, threshold, source, hint) } // sourceMovedUnderTheRows decides whether the re-payment projection has to reproduce the rendered content // hashes at all. // // The question it answers is «could any rendered byte have moved since the stored rows were written», and // the stored manifest answers it: its validity key folds the source SHA, the encoding, the chunker // version, the segmentation budget, the language pack and the embedded data — every input the rendered // text is a function of. A manifest that still validates proves the source under those rows is the source // they were rendered from, and a same-snapshot row really does resume for $0. // // ⛔ IT IS ANSWERED ONCE AND CACHED, AND THAT IS CORRECTNESS, NOT SPEED. `translate` PERSISTS the manifest // before the consent gate runs (bookrun.go — the sidecar is written where the split every paid byte is // addressed against was just computed). So an un-cached probe, asked after that point, reads a sidecar // that already describes the NEW source, answers «nothing moved», and the expensive check is skipped on // the one surface where money is actually authorised: the re-payment consent gate would not fire on an // in-place source edit at all. The first version of this fix had exactly that hole — right on the read // path, dead on the money path — and the acceptance found it. `noteSourceVintage` is what freezes the // answer while the sidecar still describes the rows. // // Caching also removes the cost the un-cached form had: it was asked PER ROW, and each ask re-read the // sidecar and re-hashed the whole source — O(rows × bytes) inside a $0 read command. // // The direction of every failure is «assume it moved»: no manifest, an unreadable one, one written in // another document version, one whose key no longer matches — all mean the projection must do the // expensive check rather than promise $0. That is the safe direction for money: over-counting a // re-payment makes an operator consent to more than will be spent, under-counting charges him for work he // was told was free. func (r *Runner) sourceMovedUnderTheRows() bool { if r.rowsSourceMoved == nil { moved := r.loadManifest() == nil r.rowsSourceMoved = &moved } return *r.rowsSourceMoved } // noteSourceVintage freezes the answer above while the stored manifest still describes the SOURCE THE // STORED ROWS WERE WRITTEN UNDER. It must be called before anything rewrites that sidecar. func (r *Runner) noteSourceVintage() { _ = r.sourceMovedUnderTheRows() } // bankRoleCommittedUSD is what the book has already paid for its bank roles — the money that lives in // checkpoints under the synthetic terminology stage and in no chunk_status row. // // A read failure degrades to zero and says so: the projection is a decision aid, and it must not be able // to stop a $0 read. Silence would be the defect, so the WARN says what the number is missing. func (r *Runner) bankRoleCommittedUSD() float64 { var total float64 for _, role := range []string{roleTerminologist, roleClassifier} { usd, err := r.Store.RoleSpentUSD(r.Book.BookID, role) if err != nil { r.Log.Warn("the bank-role spend could not be read; the book projection is missing that whole class of money, and is therefore a LOWER bound with an unknown gap rather than with a named one", "book", r.Book.BookID, "role", role, "err", err) continue } total += usd } return total }