447 lines
28 KiB
Go
447 lines
28 KiB
Go
package pipeline
|
||
|
||
import (
|
||
"context"
|
||
"errors"
|
||
"fmt"
|
||
"time"
|
||
|
||
"textmachine/backend/internal/chunk"
|
||
"textmachine/backend/internal/config"
|
||
"textmachine/backend/internal/llm"
|
||
"textmachine/backend/internal/store"
|
||
)
|
||
|
||
// cutcall.go: the money and the disposition of a call THE ENGINE ITSELF cut short (backlog row 360;
|
||
// the owner's word on the amount is D39.230 п.1). The transport half — how a cut is detected and told
|
||
// apart from an undelivered request — lives in internal/llm/attemptcut.go; this file decides what it
|
||
// COSTS and what happens to the chunk.
|
||
//
|
||
// A delivered request is generated whether or not we are still on the line. Three things end our side
|
||
// of it, they share the money, and they differ in everything else:
|
||
//
|
||
// attempt_timeout our deadline fired mid-generation. 1 call, flagged, NOT retried. Resume: 0 calls.
|
||
// cancelled a person stopped the run. 1 call, marked `cancelled`. Resume: 1 call, SAME budget.
|
||
// connection_lost the socket broke after delivery. 2 calls, then an infra pause. Resume: 1 call, SAME budget.
|
||
//
|
||
// Until this file existed all three arrived at a branch commented «No 2xx ever arrived: nothing was
|
||
// billed», which released the reservation and booked ZERO. On the provider that ships today that
|
||
// comment is false by construction: DeepSeek answers 200 with empty lines while a request waits to be
|
||
// scheduled, so a status line says nothing about whether a generation was bought.
|
||
|
||
// cutCall is the identity of the call being settled — the same tuple runAttempt already holds, passed
|
||
// as one value because thirteen positional arguments is how a chapter index and a chunk index end up
|
||
// swapped without the compiler noticing.
|
||
type cutCall struct {
|
||
stage config.Stage
|
||
chunk chunk.Chunk
|
||
job *store.Job
|
||
model string
|
||
reqHash string
|
||
resv store.Reservation
|
||
estimate float64
|
||
attempt int
|
||
escalation bool
|
||
}
|
||
|
||
// stoppedPosition is the chunk×stage a stopped run was working on. It is a struct for the reason
|
||
// cutCall is: `snapID` and `contentHash` are adjacent strings, and a swap between them compiles,
|
||
// writes a row addressed to nothing, and is found by nobody.
|
||
type stoppedPosition struct {
|
||
stage config.Stage
|
||
chunk chunk.Chunk
|
||
snapshotID string
|
||
contentHash string
|
||
cumCostUSD float64
|
||
attempts int
|
||
// paidAttempts is how many attempts of this position produced a CLASSIFICATION — a reply that was
|
||
// paid for (freshly, or on an earlier run and replayed from its checkpoint) and judged.
|
||
//
|
||
// ⛔ IT IS NOT `attempts`, AND THE DIFFERENCE IS A UNIT. `attempts` follows the attempt INDEX, which
|
||
// walks over burned keys and includes the index a reservation was refused at — so after a stopped run
|
||
// the first fresh purchase of a position happens at index ≥ 1 with nothing classified at all. A mark
|
||
// keyed on the index would write a verdict about a unit nobody has translated once.
|
||
paidAttempts int
|
||
// inHand is the last attempt that WAS classified: its index and what it answered. The ceiling mark
|
||
// below needs both, because its own reason says why the re-attack never happened and nothing else on
|
||
// the row would then say what the paid attempt came back as.
|
||
inHand stageAttempt
|
||
// firstFlagReason is what the FIRST attempt of this position failed with — the telemetry column
|
||
// (chunk_status.first_flag_reason) every row whose own verdict is no longer that failure carries.
|
||
//
|
||
// ⛔ A MARK THAT DROPPED IT WOULD BLIND THE ECHO METRIC ON A RUN THAT PAID FOR AN ECHO. The metric
|
||
// counts a draft whose first attempt echoed through `flag_reason OR first_flag_reason` (quality.go):
|
||
// a marked row's flag_reason says why the PURCHASE did not happen, so without this column the unit
|
||
// stays in the denominator and leaves the numerator, and the echo rate FALLS on the run that bought
|
||
// the echo — the 25.07 defect («echo_draft=0.0% of 20») one column further along.
|
||
firstFlagReason FlagReason
|
||
}
|
||
|
||
// settleUSDForCutCall is the ONE place that answers «what does a call we cut short cost».
|
||
//
|
||
// The answer is the reservation's own estimate (D39.230 п.1). Nobody knows what the provider actually
|
||
// billed, and whether it bills a cut call at all is deliberately OPEN — the paid balance probe that
|
||
// would answer it was declined — so the estimate is an upper bound under the assumption that the
|
||
// vendor charges for what it generated. Booking zero understates the book against its own ceiling and
|
||
// hands the platform a margin it cannot measure.
|
||
//
|
||
// It is a function and not a YAML field because a money policy behind a config knob can be changed by
|
||
// an operator who is not deciding a money policy, silently, between two runs of one book — leaving the
|
||
// ledger with two answers to one question and nothing on the row saying which.
|
||
func settleUSDForCutCall(estimate float64) float64 { return estimate }
|
||
|
||
// settleCutCall books the money for a delivered call the engine cut short, records the row, and
|
||
// returns the disposition that cause deserves.
|
||
//
|
||
// The money is written FIRST, before anything decides what happens to the chunk: the provider's side
|
||
// of the transaction already happened, and a process that dies between here and the verdict must leave
|
||
// the spend recorded. The checkpoint is what makes booking it possible at all — the store cannot write
|
||
// spend without one — and for two of the three causes it is also what stops the next run from buying
|
||
// the same call again.
|
||
func (r *Runner) settleCutCall(ctx context.Context, c cutCall, cut *llm.AttemptCutError, err error, att stageAttempt) (stageAttempt, error) {
|
||
// ⛔ THE MONEY IS DRAWN ON THE PROVIDER'S OWN ACKNOWLEDGEMENT, not on our write. A request whose
|
||
// bytes entered the peer's TCP window may never reach the application behind it — a load balancer
|
||
// accepts, the backend never sees it — and settling for that charges a reader for a call nobody
|
||
// ran, which is the one direction the canon forbids (D39.196 п.2а). A 2xx object in our hands is
|
||
// the acknowledgement; without it the same checkpoint is written for ZERO.
|
||
//
|
||
// The row is identical either way: same key, same class, same flag, same resume. Only the number
|
||
// and the `estimated` mark move, so an operator sees the class and the reader pays only for what a
|
||
// provider confirmed taking.
|
||
cost := 0.0
|
||
if cut.Billable {
|
||
cost = settleUSDForCutCall(c.estimate)
|
||
}
|
||
finish := string(cut.Cause)
|
||
if serr := r.Store.SettleWithCheckpoint(c.resv, cost, store.Checkpoint{
|
||
RequestHash: c.reqHash, JobID: c.job.ID, ChunkIdx: c.chunk.ChunkIdx, Attempt: c.attempt,
|
||
Stage: c.stage.Name, Role: c.stage.Role, ModelRequested: c.model, ModelActual: c.model,
|
||
ResponseText: "", UsageJSON: "{}", CostUSD: cost, FinishReason: finish,
|
||
Escalation: c.escalation,
|
||
}, r.events.spendLine()); serr != nil {
|
||
// The rule of every settle on this path: money state falling behind reality is a loud infra
|
||
// fault, never something to continue on top of.
|
||
return att, fmt.Errorf("pipeline: settle after a %s cut a delivered call: %w", finish, serr)
|
||
}
|
||
r.events.flush()
|
||
att.finish = finish
|
||
// `att` arrives carrying the money of any burned keys runAttempt walked over, and that money is
|
||
// this chunk's just as much as this call's — dropping it would under-state a position that was cut
|
||
// twice. `runCost` takes only what THIS run bought.
|
||
att.cumCost, att.runCost = att.cumCost+cost, cost
|
||
|
||
rl := r.baseRequestLog(c.stage, c.chunk, c.model, c.reqHash)
|
||
rl.CostUSD, rl.LatencyMS, rl.FinishReason = cost, att.latency, finish
|
||
rl.Degraded, rl.Err, rl.OK = finish, err.Error(), false
|
||
if cut.Deliveries > 1 {
|
||
// ⚠ THE LEDGER IS SHORT BY DESIGN HERE, AND THE ROW SAYS SO. The provider was asked to generate
|
||
// more than once inside one retry chain, and the store books spend only through a checkpoint —
|
||
// they share one key, so one settle covers all of them. Under-counting is the ratified
|
||
// direction (D39.196 п.2а: the deploy absorbs it, the user's balance is never overstated), so
|
||
// this pack does NOT quietly multiply the charge; it makes the gap readable instead, because a
|
||
// silent under-count is what row 360 was opened about.
|
||
// ⛔ THE NOTE GOES FIRST, and that is not cosmetics. The column an operator reads is bounded —
|
||
// `errTail` keeps 120 bytes — and the transport error in front of it is routinely longer than
|
||
// that, so a note appended to the end reached nobody, ever. It also says what was ACTUALLY
|
||
// booked: on a cut the provider never acknowledged, the answer is nothing, and a line claiming
|
||
// «one estimate is booked» beside a $0 row is a sentence that has to be disbelieved to be used.
|
||
rl.Err = cutErrLine(err, cut.Deliveries, cost)
|
||
}
|
||
// «Never write a silent $0 usage; estimate and FLAG it» is the one discipline every surveyed
|
||
// harness converges on (research/21 §5.3: crush `EstimatedUsage`, goose `CostSource`). The flag is
|
||
// what the operator's legend, the machine-readable pair beside committed_usd and the platform's
|
||
// «≥ X, up to Y» all read.
|
||
rl.Estimated, rl.EstTokens = cost > 0, r.estOutTokens(c.chunk.Text)
|
||
r.Store.LogRequest(ctx, r.Log, rl)
|
||
|
||
r.Log.WarnContext(ctx, "we cut a delivered call; the reservation estimate is charged as an ESTIMATE only when the provider had acknowledged it with a reply",
|
||
"stage", c.stage.Name, "chapter", c.chunk.Chapter, "chunk", c.chunk.ChunkIdx, "attempt", c.attempt,
|
||
"cause", finish, "after_headers", cut.AfterHeaders, "bytes_read", cut.BytesRead,
|
||
"whitespace_only", cut.WhitespaceOnly, "elapsed", cut.Elapsed.Round(time.Millisecond).String(),
|
||
"deliveries", cut.Deliveries, "estimate_usd", fmt.Sprintf("%.6f", cost))
|
||
|
||
if cut.Cause == llm.CutBySelfDeadline {
|
||
// Our deadline, the run is healthy: a disposition, not an infra failure. Nothing retries it —
|
||
// a retry asks the provider to generate, and bill, the very thing it is generating right now.
|
||
att.cls = classification{FlagAttemptTimeout, cut.Error()}
|
||
r.setJobStatus(ctx, c.job.ID, "done")
|
||
return att, nil
|
||
}
|
||
// A stopped run and a dead socket both end this stage. The money above is recorded either way; the
|
||
// error travels so the run pauses instead of walking the rest of the book against a dead provider.
|
||
//
|
||
// ⛔ BUT THE TWO DO NOT GET THE SAME WORD (backlog row 389, and this site is not one of the two the row
|
||
// names — it is the one a stop actually goes through). A call a person cut was healthy when they cut
|
||
// it; a dead socket IS a failure and keeps the word. The question is asked once, for the whole family
|
||
// — see noteJobFailed.
|
||
r.noteJobFailed(ctx, c.job.ID, err)
|
||
return att, fmt.Errorf("pipeline: stage %s call (ch%d/chunk%d, model %s): %w",
|
||
c.stage.Name, c.chunk.Chapter, c.chunk.ChunkIdx, c.model, err)
|
||
}
|
||
|
||
// cutErrLine is the sentence an operator reads when one retry chain asked the provider to generate more
|
||
// than once. It is a named function rather than a Sprintf at the call site because both of its
|
||
// properties are load-bearing and neither is visible from there.
|
||
//
|
||
// ⛔ THE NOTE COMES FIRST. The column that prints this is bounded (cmd/tmctl's errTail keeps 120 bytes)
|
||
// and the transport error is routinely longer, so a note appended at the end reached nobody — while
|
||
// being the only place the gap between what the provider generated and what the ledger booked is
|
||
// visible at all.
|
||
//
|
||
// ⛔ AND IT SAYS WHAT WAS ACTUALLY BOOKED. On a cut the provider never acknowledged, the answer is
|
||
// nothing; «one estimate is booked» printed beside a $0 row is a sentence a reader has to disbelieve
|
||
// before they can use it.
|
||
func cutErrLine(err error, deliveries int, cost float64) string {
|
||
booked := "ONE estimate is booked for all of them"
|
||
if cost <= 0 {
|
||
booked = "NOTHING is booked: the provider acknowledged none of them"
|
||
}
|
||
return fmt.Sprintf("[the provider was asked %d times; %s] %s", deliveries, booked, err.Error())
|
||
}
|
||
|
||
// stopMark is the row a stopped run leaves on the position it was working on: its reason, the sentence a
|
||
// person reads afterwards, and how many attempts that position actually made.
|
||
type stopMark struct {
|
||
reason FlagReason
|
||
detail string
|
||
attempts int
|
||
// firstFlag is the superseded first failure, written exactly as the ok path writes it
|
||
// (recoveredFirstFlag): empty when the row's own verdict IS that failure, so nothing is counted twice.
|
||
firstFlag string
|
||
}
|
||
|
||
// stopMarkFor decides WHETHER a stop leaves a mark on this position and WHICH one. It is a pure function
|
||
// of the position and the error so the decision can be asked directly in a test, rather than only through
|
||
// a run that has to be arranged to stop in the right way.
|
||
//
|
||
// TWO stops leave a mark, and they are different facts about the same shape — money was spent on this
|
||
// position and the work behind it was not finished:
|
||
//
|
||
// - a HUMAN stopped the run over a call that was already on the wire (`cancelled`). The call was
|
||
// healthy, so the mark reads «stopped; the resume re-does it on the same budget»;
|
||
// - a USD CEILING refused to reserve the RE-ATTACK of an attempt this position had already paid for
|
||
// (`retry_unaffordable`, backlog row 291). The run stops like any other ceiling halt — that part is
|
||
// ratified and unchanged — but the position used to leave NO row at all, so the unit read `pending`,
|
||
// indistinguishable from one nobody had started, while its first attempt was paid for and on disk.
|
||
// Measured on the fixture of retrystopmark_test.go before this existed: `committed=$0.003640` with
|
||
// ONE chunk_status row for the two units the run had touched.
|
||
//
|
||
// ⛔ EVERY OTHER STOP LEAVES NOTHING, and that is a statement rather than an omission. A ceiling that
|
||
// refuses attempt 0 has bought nothing this position can DELIVER: `pending` is then the truth about the
|
||
// TEXT, and a row would invent a half-done unit. That is why the ceiling branch asks `paidAttempts`, not
|
||
// the attempt index.
|
||
//
|
||
// ⚠ «NOTHING IT CAN DELIVER» IS NARROWER THAN «NOTHING», and the difference is a door this pack leaves
|
||
// open: a position whose only purchase was a BURNED key (money, no result — burnedByCut) and whose next
|
||
// index a ceiling refused has `committed > 0` with no row, which is the same invisible shape this mark
|
||
// exists to end, minus any text to account for. Closing it needs a reason of its own («paid for, nothing
|
||
// came back, nothing translated») or the owner's word that a burn is visible through the ledger alone —
|
||
// and the table of stopMarkFor pins today's answer so the next reader finds a decision, not a gap.
|
||
//
|
||
// A flag lying about its cause is forbidden in its own right (D39.93 п.2), which is why the second case
|
||
// gets a reason of its own instead of borrowing `cancelled`: nobody stopped that run by hand.
|
||
func stopMarkFor(p stoppedPosition, err error) (stopMark, bool) {
|
||
// ⛔ THE STOP IS ASKED OF THE ERROR AS A WHOLE, and the delivery of ANY cut in it — not of whichever
|
||
// cut `errors.As` happens to reach first. A retry chain hands up the cut with the strongest money
|
||
// claim (llm's chainError), which is deliberately the EARLIER one when a later free cut would mask
|
||
// it — so a run stopped over an attempt behind a paid `connection_lost` arrives here carrying that
|
||
// cause, and a guard reading only the first cut's cause decided «this is not a stop» and returned in
|
||
// silence. The money was settled, the position got no row, and the export showed an unexplained gap:
|
||
// the hole §4.2 forbids «at any moment».
|
||
// ⚠ «A cut» IS «a delivered cut», by construction and in one place: cutError returns NIL when the
|
||
// request never went out, and the single construction of the type sets Delivered true. So one
|
||
// errors.As answers both halves, and a walk over the error tree looking for a delivered one would be
|
||
// asking a question that cannot come back different.
|
||
var cut *llm.AttemptCutError
|
||
if errors.Is(err, context.Canceled) && errors.As(err, &cut) {
|
||
return stopMark{
|
||
reason: FlagCancelled,
|
||
detail: "the run was stopped while this call was in flight; it was paid for at the reservation estimate and the resume re-does it on the same budget",
|
||
attempts: p.attempts,
|
||
firstFlag: recoveredFirstFlag(p.firstFlagReason, FlagCancelled),
|
||
}, true
|
||
}
|
||
if p.paidAttempts > 0 && errors.Is(err, errReserveCeiling) {
|
||
return stopMark{
|
||
reason: FlagRetryUnaffordable,
|
||
detail: ceilingStopDetail(p, err),
|
||
// ⚠ ONE FEWER THAN `attempts`, AND THE BRANCH ABOVE DOES NOT SUBTRACT — the two stops differ
|
||
// in exactly this. `attempts` is the index the loop is ON (attemptsMade = attempt + 1, set
|
||
// before the error check), so for a cancelled call it counts the call that DID go out, while
|
||
// here the index it counts bought nothing at all. What is left is every index this position
|
||
// really consumed, burned keys included — the same thing the ok path's count includes, so a
|
||
// row cut from `paidAttempts` instead would silently drop a burn this position paid for.
|
||
//
|
||
// ⚠ IT CANNOT GO NEGATIVE, and the reason is not local: `paidAttempts > 0` above means the loop
|
||
// classified something, and the loop sets attemptsMade = attempt + 1 ≥ 1 before any error is
|
||
// read (stagerun.go). A future shape that marked a position without that guarantee would have
|
||
// to bring the floor with it.
|
||
attempts: p.attempts - 1,
|
||
// ⚠ BOTH MARKS CARRY IT, and the cancelled one did not until this pack: a position stopped over
|
||
// its SECOND attempt has a first failure too, and the same blinding applied to it. One rule for
|
||
// the two stop marks rather than a rule and an exception.
|
||
firstFlag: recoveredFirstFlag(p.firstFlagReason, FlagRetryUnaffordable),
|
||
}, true
|
||
}
|
||
return stopMark{}, false
|
||
}
|
||
|
||
// ceilingStopDetail is the sentence the money mark leaves behind. It carries three things the row cannot
|
||
// get anywhere else: WHAT the paid attempt answered (the reason column now says why the re-attack never
|
||
// happened, not what came back), WHICH ceiling refused, and HOW MUCH was missing. The log line says the
|
||
// same, and dies with the process; the row is what a person reads the next morning.
|
||
//
|
||
// ⚠ IT NAMES THE RE-ATTACK, and that is only true while every OTHER paid step of a position degrades on a
|
||
// ceiling instead of propagating it: the escalation hop, the repair sub-step and the terminology pass all
|
||
// catch errReserveCeiling (escalation.go, repair.go, terminologist.go), so the one that reaches here is
|
||
// the attempt loop's. A new paid sub-step that let a ceiling through would make this sentence misname
|
||
// which purchase was refused.
|
||
func ceilingStopDetail(p stoppedPosition, err error) string {
|
||
// WHICH ceiling is named by the typed stop, and the whole noun phrase is built here rather than left to
|
||
// a placeholder inside the sentence: a bare «%s» with a defensive default produced «the a USD ceiling»
|
||
// on the day the type was not there, and this line is what a person reads the next morning.
|
||
//
|
||
// ⚠ THE DEFAULT IS UNREACHABLE TODAY, and the condition is worth naming rather than trusting: both
|
||
// wrappers of errReserveCeiling are inside a *CeilingHalt (stagerun.go), so an `errors.Is` that holds
|
||
// cannot sit on an `errors.As` that fails. It becomes reachable the day something wraps that sentinel
|
||
// without the type — and then the sentence still reads.
|
||
ceiling, shortfall := "a USD ceiling", int64(0)
|
||
var halt *CeilingHalt
|
||
if errors.As(err, &halt) {
|
||
ceiling, shortfall = "the "+halt.Scope+" USD ceiling", halt.ShortfallMicroUSD
|
||
}
|
||
// «Short by N» is omitted rather than printed as zero when the stop did not state it (a day-scope
|
||
// refusal, or a ledger read that failed — shortfallMicroUSD): a money figure of 0 reads as «you are
|
||
// not short of anything», which is the opposite of what happened.
|
||
missing := ""
|
||
if shortfall > 0 {
|
||
missing = fmt.Sprintf(", short by %d micro-USD", shortfall)
|
||
}
|
||
return fmt.Sprintf("attempt %d was paid for and came back %s; %s refused to reserve the re-attack%s — raise the ceiling and the resume finishes this unit",
|
||
p.inHand.attempt, p.inHand.cls.Reason, ceiling, missing)
|
||
}
|
||
|
||
// recordStoppedPosition marks a position a stopped run was working on, so the stop leaves no unexplained
|
||
// gap. It never changes the outcome it is called on: the run is ending, and this says what it was
|
||
// doing when it did. WHICH stops leave a mark, and why, is stopMarkFor above.
|
||
//
|
||
// A write failure is logged, not returned: the caller is already returning the error that stopped the
|
||
// run, and replacing it with a bookkeeping failure would hide why the run stopped.
|
||
func (r *Runner) recordStoppedPosition(ctx context.Context, p stoppedPosition, err error) {
|
||
mark, marked := stopMarkFor(p, err)
|
||
if !marked {
|
||
return
|
||
}
|
||
// ⛔ A CUT MUST NOT BLIND A VERDICT IT DID NOT PRODUCE, and this row is an UPSERT: chunk_status is
|
||
// keyed (book, chapter, chunk, stage), so the write below REPLACES whatever the position already held
|
||
// — including `final_hash`, which is the only pointer to the text a previous run bought and the
|
||
// export ships.
|
||
//
|
||
// ⚠ IT IS REACHABLE, and it was measured rather than argued (backlog row 375, the ⛔ half): a prompt
|
||
// edit moves both the rendered content and the snapshot, so the resume fast-path above refuses the
|
||
// stored row and the position is re-attacked; cut the run there and the row went from
|
||
// `ok / final_hash=770e5563 / cost=$0.001820` to `flagged(cancelled) / final_hash="" / cost=$0`, the
|
||
// checkpoint holding the text stayed on disk with nothing addressing it, and the export shipped an
|
||
// empty chapter where it had shipped prose. The run made the book WORSE than it found it, which is
|
||
// what D39.240 forbids of a stop: it may cost money, it may not leave rubbish.
|
||
//
|
||
// So the mark is written only where it was meant to be written — over a position that has no verdict
|
||
// of its own — and a position that already holds one keeps it. Nothing is hidden by that: the resume
|
||
// re-attacks this position anyway (its stored content hash no longer matches, which is why it was
|
||
// re-attacked in the first place), the cut call's money is on the ledger through its checkpoint and
|
||
// its request_log row, and the line below says out loud what was kept and why.
|
||
//
|
||
// ⚠ THE REDRIVE ROUTE CANNOT REACH THIS: ResetChunkStages deletes the row and its checkpoints in one
|
||
// transaction before re-attacking, so there is nothing to protect and this guard is a no-op there.
|
||
if prev, perr := r.Store.GetChunkStatus(r.Book.BookID, p.chunk.Chapter, p.chunk.ChunkIdx, p.stage.Name); perr != nil {
|
||
// Not fatal and not silent: the caller is already returning the error that stopped the run, and a
|
||
// failed read here only means the guard cannot answer — so it does not, and the mark is written as
|
||
// it always was.
|
||
r.Log.WarnContext(ctx, "could not read the position's stored disposition before marking it stopped; the mark is written and may replace an earlier verdict",
|
||
"stage", p.stage.Name, "chapter", p.chunk.Chapter, "chunk", p.chunk.ChunkIdx, "err", perr)
|
||
} else if prev != nil && resolvedForResume(prev) {
|
||
r.Log.WarnContext(ctx, "the run was stopped over a position that ALREADY held a verdict from an earlier run; that verdict and its text are kept rather than overwritten by the stop mark (the resume re-does this position, and the stopped call's money is on the ledger)",
|
||
"stage", p.stage.Name, "chapter", p.chunk.Chapter, "chunk", p.chunk.ChunkIdx,
|
||
"kept_disposition", prev.Disposition, "kept_flag_reason", prev.FlagReason,
|
||
"stopped_call_cost_usd", fmt.Sprintf("%.6f", p.cumCostUSD))
|
||
return
|
||
}
|
||
if uerr := r.Store.UpsertChunkStatus(store.ChunkStatus{
|
||
BookID: r.Book.BookID, Chapter: p.chunk.Chapter, ChunkIdx: p.chunk.ChunkIdx, Stage: p.stage.Name,
|
||
SnapshotID: p.snapshotID, ContentHash: p.contentHash,
|
||
Disposition: string(DispFlagged), FlagReason: string(mark.reason),
|
||
Attempts: mark.attempts, FinalHash: "", CostUSD: p.cumCostUSD,
|
||
Detail: mark.detail, FirstFlagReason: mark.firstFlag,
|
||
}); uerr != nil {
|
||
r.Log.ErrorContext(ctx, "could not mark the stopped position; its money is recorded but the chunk will read as never started",
|
||
"stage", p.stage.Name, "chapter", p.chunk.Chapter, "chunk", p.chunk.ChunkIdx, "err", uerr)
|
||
return
|
||
}
|
||
if mark.reason == FlagRetryUnaffordable {
|
||
r.Log.WarnContext(ctx, "re-attack denied by a USD ceiling; the position is marked paid-but-unfinished so it does not read as never started, and a raised ceiling finishes it on resume",
|
||
"stage", p.stage.Name, "chapter", p.chunk.Chapter, "chunk", p.chunk.ChunkIdx,
|
||
"paid_attempt", p.inHand.attempt, "paid_reason", string(p.inHand.cls.Reason),
|
||
"cost_usd", fmt.Sprintf("%.6f", p.cumCostUSD), "detail", mark.detail)
|
||
return
|
||
}
|
||
r.Log.WarnContext(ctx, "the run was stopped over a call that had already gone out; the position is marked cancelled and the resume re-does it on the same budget",
|
||
"stage", p.stage.Name, "chapter", p.chunk.Chapter, "chunk", p.chunk.ChunkIdx, "cost_usd", fmt.Sprintf("%.6f", p.cumCostUSD))
|
||
}
|
||
|
||
// paidAfterBurns answers «was this call already paid for AND answered», asked THE WAY THE FUNNEL WILL
|
||
// ASK IT. It is the single definition of that question for every pre-gate that decides money before a
|
||
// call, and it exists because asking it any other way has now been wrong in both directions.
|
||
//
|
||
// ⛔ A FIXED ATTEMPT INDEX CANNOT ANSWER IT. runAttempt does not stop at a burned key: it walks to the
|
||
// next index at the SAME budget and buys there, so after a stopped run a position reads «attempt 0
|
||
// burned, attempt 1 paid and answered». A probe that looks only at the starting index sees the burn and
|
||
// says «not paid» — and the caller, finding no budget left, discards a translation that was already
|
||
// bought. A probe that ignores burns says «paid» — and the caller skips its budget check while the
|
||
// funnel goes and buys the work again. Both were measured, one after the other, on this very code.
|
||
//
|
||
// So the probe walks exactly as the funnel walks (stagerun.go, the burn loop) and answers about the key
|
||
// the funnel will actually use. Two questions asked one way cannot disagree; that is the whole point of
|
||
// this function existing rather than three call sites each getting the walk right.
|
||
func (r *Runner) paidAfterBurns(st config.Stage, model, snapID string, ch chunk.Chunk, attempt, maxTokens int, msgs []llm.Message) (bool, error) {
|
||
for {
|
||
cp, err := r.Store.GetCheckpoint(RequestHash(r.attemptRequest(st, model, snapID, ch, attempt, maxTokens, msgs)))
|
||
if err != nil {
|
||
return false, err
|
||
}
|
||
if cp == nil {
|
||
return false, nil // the walk ran out of keys: nothing here is paid for
|
||
}
|
||
if !burnedByCut(cp) {
|
||
return true, nil // an answer — this is what the funnel would serve for $0
|
||
}
|
||
attempt++
|
||
}
|
||
}
|
||
|
||
// resolvedForResume says a stored disposition is an ANSWER the resume may serve without calling
|
||
// anybody. Everything terminal is; two rows are not, and they are not for the same reason — each
|
||
// records something that was never DONE rather than a verdict about what came back:
|
||
//
|
||
// - `cancelled` — a person stopped the run over a call that was already on the wire;
|
||
// - `retry_unaffordable` — the first attempt was paid for and flagged, and a ceiling refused to
|
||
// reserve the retry. It is the one flag money CURES (D4: raise the ceiling, resume, the reader gets
|
||
// a good translation), so a resume must re-attack it rather than serve the shortfall as an answer.
|
||
// Re-attacking costs nothing while the ceiling stands: attempt 0 replays from its checkpoint for $0
|
||
// and the refusal recurs, which is why the re-attack is safe to repeat on every resume.
|
||
//
|
||
// Reading either as terminal degenerates the whole construction into its opposite — never re-doing
|
||
// anything that was interrupted — which is the failure mode this design is most at risk of, since it
|
||
// would look perfectly green.
|
||
func resolvedForResume(cs *store.ChunkStatus) bool {
|
||
return FlagReason(cs.FlagReason).AnswersForResume()
|
||
}
|
||
|
||
// ResolvedForResume is the same question for the CLI, which has to split a book's rows into «answers the
|
||
// next run serves for $0» and «positions it will do again».
|
||
//
|
||
// ⛔ IT IS EXPORTED RATHER THAN RE-WRITTEN THERE, and the reason is a defect that existed for the length
|
||
// of one edit: the stopped-run account had the split spelled as `FlagReason == FlagCancelled`, a second
|
||
// copy of this rule — and the day a SECOND non-resolved reason existed (retry_unaffordable) that copy
|
||
// started counting a position the resume re-does as one with a verdict, telling an operator the rest is
|
||
// served for $0. One predicate, asked by both surfaces.
|
||
func ResolvedForResume(cs *store.ChunkStatus) bool { return resolvedForResume(cs) }
|