package pipeline import ( "context" "math" "sync" "textmachine/backend/internal/store" ) // reservegate.go: the money admission, as a gate the whole run shares instead of a call each worker // makes alone (backlog row 277). // // ⛔ THE DEFECT THIS EXISTS FOR. A reservation is refused when `committed + reserved + estimate` would // pass the ceiling (store/ledger.go). Under W parallel workers the refusal reached runWave as an ordinary // error, runWave cancelled the derived context, and every call ALREADY ADMITTED died with it — so a book // whose ceiling could not admit the W'th call produced NOTHING, not even the W-1 calls the money was // there for. Measured live 04.09 and reproduced twice: the run departed in 10-15 seconds having moved the // book by zero units. The cure is not a bigger ceiling — the wall stands wherever the cheapest ceiling is // smaller than W indivisible steps — it is that a refusal must mean «do not START anything new» rather // than «kill what is running». The volume ceiling in this same package was already built that way and // says so in its own words: «THE STOP IS A COMPLETION, NOT A HALT» (volume.go), and it names the reason a // worker-side counter is the wrong shape — «a worker refused a slot would have to abandon its item // permanently». // // ⚠ THE COUNTER IS NOT AN OPTIMISATION, IT IS THE THING THAT MAKES A REFUSAL NON-FINAL. Settle replaces a // reservation with the ACTUAL cost, and the actual cost is never more than the estimate (ledger.EstimateUSD // is deliberately pessimistic: worst-case cache write, completion running to max_tokens). So while any // call is in flight the headroom can only GROW, and a refusal taken while the sky is full is a refusal // about a number that is still moving. Only a refusal with nothing in flight is about a number that has // stopped — and only that one stops the book. // // ⚠ RESERVE RUNS UNDER THIS MUTEX, and that is a correctness requirement rather than tidiness. With the // admission and the counter in two critical sections, worker X could commit its reservation and be // pre-empted before recording itself as in flight; worker Y, refused BY X's reservation, would then read // zero in flight and declare the book finished while X's call was running and about to give money back. // The store already serialises reservations through its single write pool, so nothing is lost by holding // them under one more lock. type reserveGate struct { mu sync.Mutex // inFlight is the number of calls that hold a live reservation — admitted, not yet settled or // released. It is the ONLY question the gate answers: «can the headroom still grow?». inFlight int // settled is broadcast: closed and replaced every time a reservation ends. A waiter captures it under // the same lock it reads inFlight under, so a settle that lands between the read and the wait closes // the channel the waiter is already holding, and no wake-up can be missed. settled chan struct{} // waiting is how many callers are queued for headroom right now. It is OBSERVABILITY — the log line // an operator reads when a wave has gone serial says how many workers are stacked behind the last // dollars — and it decides nothing: a queue is not a claim about whether money will free up. waiting int // optionalTurnedAway counts optional callers refused WHILE money was still moving — the exact event // that discriminates the rule. In a correct run it grows and `waits` does not; if an optional site // were ever handed `mandatory: true`, the two swap. optionalTurnedAway int // refusals counts admissions the ceiling turned down, whatever happened next. It is what lets a test // open the window this whole pack is about — «a refusal HAS happened and a call is STILL in flight» — // without guessing at timing. refusals int // waits counts how many times a caller actually WAITED AND WAS WOKEN — the mechanism firing, not the // queue's depth. // // ⚠ IT EXISTS BECAUSE A TEST CANNOT SEE THE MECHANISM ANY OTHER WAY, and a test that cannot see it // asserts the wrong thing. The first version of the delivering-ceiling test asserted only the OUTCOME // («the book was delivered»), and it passed on a fixture whose ceiling never refused anything at all: // the wait was not exercised, the mutant that deletes the wait would have survived it, and the // assertion read as proof of a path the run never took. Executed proof of a mechanism needs a fact // the mechanism leaves behind; this is it. waits int } // admit performs the reservation and records the caller as in flight if it was granted. Every caller that // gets ReserveOK MUST call done() exactly once — as a defer at the call site, never as a set of calls on // the exit paths, because a single missed one leaves inFlight above zero for the rest of the run and // every later refusal then waits for a settle that will not come. func (g *reserveGate) admit(s *store.Store, bookID string, estimate float64, c store.Ceilings) (store.Reservation, store.ReserveResult, error) { g.mu.Lock() defer g.mu.Unlock() resv, verdict, err := s.Reserve(bookID, estimate, c) switch { case err != nil: case verdict == store.ReserveOK: g.inFlight++ default: g.refusals++ } return resv, verdict, err } // done records that a reservation has ended — settled or released, the gate does not care which, because // both give headroom back (settle gives back estimate−cost, release gives back the whole estimate). func (g *reserveGate) done() { g.mu.Lock() defer g.mu.Unlock() if g.inFlight > 0 { g.inFlight-- } if g.settled != nil { close(g.settled) g.settled = nil } } // waitOutcome is why a wait ended, and it has THREE values because two of them are opposite facts that a // boolean made identical. // // ⚠ THE MISTAKE THIS TYPE EXISTS TO PREVENT WAS MADE HERE ONCE. With a bool, «nothing is in flight» and // «the context died» both read false, and the caller turned both into a CeilingHalt. So a run stopped by // Ctrl-C, or by a sibling's infra failure cancelling the context, published a `ceiling` event with a // shortfall and departed telling the platform to add money — for a stop that had nothing to do with // money. That is exactly the class the two-slot error ranking in runWave was built to rule out, arriving // through the waiting path instead. type waitOutcome int const ( // waitSettled: a reservation ended, the headroom grew, ask again. waitSettled waitOutcome = iota // waitNothingInFlight: no call holds a reservation, so the number the refusal was taken against has // stopped moving. THIS is the refusal that stops the book. waitNothingInFlight // waitAborted: the run is ending for a reason that is not money. The caller must surface THAT, and // must not claim a ceiling stopped anything. waitAborted // waitNotAllowed: money is still moving, but THIS caller may not queue for it — an optional call, // which degrades on a ceiling instead of stopping the book. Distinct from waitNothingInFlight because // the two are opposite facts about the run: one says the number has stopped, the other says the // caller has no claim on it. waitNotAllowed ) // waitForSettle blocks until some in-flight call gives its reservation back, and says why it stopped // waiting. See waitOutcome — the four answers lead to four different things the caller must do. // // ⛔ THE «MAY THIS CALLER WAIT AT ALL» RULE LIVES HERE, and it moved here because as an `if` at the call // site it was a rule nothing could observe. `mandatory` says whether a refusal stops the book: the run's // own wave work is mandatory, while the escalation hop, the repair sub-step and the terminology pass all // degrade on errReserveCeiling instead. An optional call that queued for the last dollars would turn «do // not start anything new» into «the optional spends what the mandatory needed» — and the hop holds escMu // for its whole life, so it would block every other worker's escalation while it waited. Asked here, the // rule leaves a counter behind and a test can see it happen. func (g *reserveGate) waitForSettle(ctx context.Context, mandatory bool) waitOutcome { g.mu.Lock() if g.inFlight == 0 { g.mu.Unlock() return waitNothingInFlight } if !mandatory { g.optionalTurnedAway++ g.mu.Unlock() return waitNotAllowed } if g.settled == nil { g.settled = make(chan struct{}) } ch := g.settled g.waiting++ g.mu.Unlock() defer func() { g.mu.Lock() g.waiting-- g.mu.Unlock() }() select { case <-ch: g.mu.Lock() g.waits++ g.mu.Unlock() return waitSettled case <-ctx.Done(): return waitAborted } } // stateNow is the gate's two counters for a log line and for a test. They are deliberately NOT used to // decide anything: a decision taken on them outside the lock is the race admit() exists to close. func (g *reserveGate) stateNow() (inFlight, waiting int) { g.mu.Lock() defer g.mu.Unlock() return g.inFlight, g.waiting } // waitsSoFar is how many times the run has waited for headroom and been woken by a settle — the count a // test asks to know that the mechanism RAN, rather than inferring it from an outcome the run could have // reached without it. func (g *reserveGate) waitsSoFar() int { g.mu.Lock() defer g.mu.Unlock() return g.waits } // optionalTurnedAwaySoFar is how many optional calls were refused while money was still moving and were // NOT allowed to queue for it — the fact that distinguishes the rule from its absence. // refusalsSoFar is how many admissions the ceiling has turned down in this run. func (g *reserveGate) refusalsSoFar() int { g.mu.Lock() defer g.mu.Unlock() return g.refusals } func (g *reserveGate) optionalTurnedAwaySoFar() int { g.mu.Lock() defer g.mu.Unlock() return g.optionalTurnedAway } // settleCannotHelp reports that no amount of waiting can admit this estimate, because the committed // spend ALONE already leaves no room for it. // // It is the difference between a slow answer and a wrong wait. A settle returns at most estimate−cost of // a reservation; it can never reduce what is already committed. So once `committed + estimate` is past // the ceiling, every in-flight call could finish for free and the answer would still be no — and a // waiter that did not ask this question would hold its wave open for the whole of the longest provider // timeout (attempt_timeout × transport retries — minutes) to arrive at the refusal it already had. // // ⚠ IT IS ASKED ONLY OF THE BOOK CEILING, and the asymmetry is honest rather than an omission. The day // ceiling sums every book sharing this store (store/ledger.go: `WHERE date = ?` with no book filter), so // this process's committed figure is not the day's, and a conservative «wait» is the only answer it can // give without inventing one. Waiting is bounded anyway: the wait ends the moment nothing is in flight, // which is the same terminal condition. func settleCannotHelp(scope string, committed, estimate, bookCeiling float64) bool { return scope == scopeBookCeiling && bookCeiling > 0 && committed+estimate > bookCeiling } // scopeBookCeiling mirrors runevents.ScopeBook without importing it into this predicate's signature — // the gate reasons about money, not about the event vocabulary. const scopeBookCeiling = "book" // shortfallMicroUSD is HOW MUCH WAS MISSING — the only money figure that leaves the engine when a ceiling // stops a run (backlog rows 277/278). // // ⚠ IT IS THE SHORTFALL AND NOT THE DENIED ESTIMATE, and the difference is the difference between a // number a person can act on and one the contract forbids. The denied estimate is the PRICE OF A CALL, // and «цены моделей, стадий и вызовов не выходят» — publishing it would put the engine's per-call cost on // a wire that D39.196 §2а deliberately keeps to balance, ceiling and hold. The shortfall says «top up by // at least this and the run continues», which is the platform's actual question, and it discloses nothing // about what any one stage costs: it is the distance between a limit the platform itself set and the // total it has already authorised. // // It mirrors the store's own admission inequality — a reservation is refused when // `committed + reserved + estimate` passes the ceiling — so the number is exactly the amount by which // that comparison failed. Integer micro-USD, rounded UP, for the reason Spend rounds up: money never // travels as a float (PD-79), and a figure a person tops up AGAINST must never be short. // // ⚠ IT IS AN UPPER BOUND, NOT A MINIMUM, and the difference is worth stating because the obvious reading // is the other one. `reserved` includes calls that are still running, and a settle replaces a reservation // with a cost that is never larger — so by the time the money arrives, some of what this figure counts // will already have been given back. «Top up by this and the refused call is admitted» is therefore // always true; «this is the least that would do» is not. When nothing was in flight — the ordinary // terminal refusal — the two coincide. // // ⚠ BOOK SCOPE ONLY, and the absence is honest rather than lazy. The daily ceiling is summed over EVERY // book sharing this store (store/ledger.go: `WHERE date = ?`, no book filter), while the committed and // reserved figures available here are this book's alone. A shortfall computed from them would be a // confident number about the wrong total, and the day ceiling is in any case one the platform neither // sets nor sees (PD-157). Zero means «not stated», and the field is omitted from the wire. func shortfallMicroUSD(scope string, committed, reserved, estimate, ceiling float64) int64 { if scope != scopeBookCeiling || ceiling <= 0 { return 0 } missing := committed + reserved + estimate - ceiling if missing <= 0 { return 0 } // ⚠ ROUNDED TO THE NANO-DOLLAR BEFORE BEING ROUNDED UP TO THE MICRO. Four float additions of prices // leave noise far below a micro-dollar — $0.30 + $0.20 + $0.07 − $0.55 evaluates to // 0.02000000000000002 — and a bare Ceil turns that dust into a whole extra micro-dollar. Rounding to // the nanodollar first discards the noise and nothing else: a real difference of one nanodollar is // not a difference in money, and everything above the micro-dollar still rounds UP. return int64(math.Ceil(math.Round(missing*1e9) / 1e3)) }