package llm import ( "context" "errors" "fmt" "net/http/httptrace" "strings" "sync/atomic" "time" ) // attemptcut.go: what OUR OWN deadline does to a call, and the deadline itself. // // A provider that has accepted a request generates whether or not we are still listening. When the // attempt deadline fires mid-generation the connection drops on our side, the provider never learns // it, and the call is billed all the same — so «we stopped waiting» is a MONEY event, not a transport // one. The transport used to lose it twice over: the body read's error went to `_`, and truncation was // judged by SIZE, so a body our own deadline cut short (small) was indistinguishable from a whole one // and went down the retryable «broken connection» branch that the same code's comment forbids for a // billed 2xx («each retry is a new billed 2xx»). // // THE ONE FACT COMMON TO EVERY SUCH CASE IS THAT THE REQUEST WAS DELIVERED, and it is observable // without a byte on the wire: net/http/httptrace reports when the request has been WRITTEN and when // the first response byte arrived. // // ⚠ NO SURVEYED HARNESS DRAWS THE BOUNDARY HERE, and that is a fact rather than a claim of novelty: // research/21 covers eleven of them and mentions httptrace nowhere (0 hits against 20 for openai-go). // The reason is what they are: five are stream-only and four do both (§Q6), so «did anything arrive» is // answered by the first event and the question never comes up; the remaining two are generic SDKs whose // non-streaming answer is to REFUSE a long call outright. On a non-streaming path with no refusal to // fall back on it has to be answered another way, and the stdlib already answers it — so the mechanism // is stdlib rather than ours (research/21's own rule: an industrial primary source before a home-made one). // // Those two booleans separate the three cases that a status line cannot: // // delivered, headers still pending, our deadline WroteRequest=true GotFirstResponseByte=false // TLS handshake hanging — NOT delivered WroteRequest=false GotFirstResponseByte=false // early 200 with empty lines, our deadline WroteRequest=true GotFirstResponseByte=true // // The middle row is the ONLY one where «nothing was bought» is true by construction. On DeepSeek the // first and third are the live ones: the vendor documents an early 200 with empty lines while a // request waits to be scheduled, so a 200 there means ACCEPTED, not GENERATED, and the branch that // reads «No 2xx ever arrived: nothing was billed» is unreachable on it — every self-cut hid under // decode_error instead. // CutCause says WHO ended an attempt that had already been delivered. The three are dispositions, // not shades of one error: our own deadline is not retried and is flagged, a run the operator // stopped is re-done on resume at the same budget, and a broken connection gets one retry. type CutCause string const ( // CutBySelfDeadline: our per-attempt deadline fired while the provider was still working. // Retrying buys the same generation a second time — the whole point of typing this. CutBySelfDeadline CutCause = "attempt_timeout" // CutByParent: the RUN ended (stop, Ctrl-C) under a call that had already gone out. The call was // healthy; a human stopped it. CutByParent CutCause = "cancelled" // CutByConnection: the connection broke after delivery (unexpected EOF, an h2 stream reset) with // both deadlines still alive. Transport-shaped, so it is worth exactly one retry. CutByConnection CutCause = "connection_lost" ) // AttemptCutError is a DELIVERED request whose reply we did not receive whole. // // ⚠ It is deliberately NOT named for the timeout: two of its three causes are not one (a run somebody // stopped, a connection that broke), and the runner routes money, flag and resume differently for each. // A type named `AttemptTimeoutError` carrying `Cause: cancelled` would be a name lying about a cause, // which is the thing the flag vocabulary forbids outright (D39.93 п.2). // // Delivered is the money boundary and is always true on a value that reaches a caller — the // constructor refuses to build one otherwise, because «not delivered» is the case that must keep // costing nothing. type AttemptCutError struct { Provider string Cause CutCause // Delivered: the request bytes reached the provider (httptrace WroteRequest, no write error). Delivered bool // AfterHeaders: the first BYTE of a response had arrived when the call was cut (httptrace's // GotFirstResponseByte) — the early-200 case. // // ⚠ IT IS A BYTE, NOT A REPLY, and the distinction is the whole reason `Billable` is not this field. // The first byte fires for a 1xx, for a header block that never terminated, for a broken proxy's // garbage — none of which is a provider acknowledging anything. Read as «a response arrived» it // tells an operator about a reply that did not exist; read as what it is, it is the evidence that // something came back down the socket, and the money question is answered by `Billable`, which also // requires a 2xx object in hand. AfterHeaders bool // BytesRead is how much of the body we did get. It is EVIDENCE, never the criterion: judging // truncation by size is the defect this type replaces. BytesRead int // WhitespaceOnly says the bytes we got carry no content at all (DeepSeek's documented empty lines // while a request waits for scheduling). It distinguishes «the provider was still queueing» from // «the provider was mid-answer», which is the difference between a wait worth extending and a // generation worth not buying twice. WhitespaceOnly bool Elapsed time.Duration // Billable is the MONEY boundary, and it is deliberately NARROWER than Delivered. // // ⛔ TWO PREDICATES, BECAUSE THEY ANSWER TWO QUESTIONS. `WroteRequest` says our bytes left for the // peer's TCP window; it says nothing about the APPLICATION behind it — a load balancer can accept // while the backend never sees the request. Measured on a stopped run: 3 of 25 cancelled in-flight // calls settled an estimate for a request no handler ever entered. On a non-streaming wire the one // signal that a provider's application acknowledged the request is a 2xx response object reaching // US, so that is what money is drawn on. Delivery still drives RETRY — a written request is not // automatically re-sent — and that boundary is unchanged. // // The direction of what remains is the tolerable one: on a provider that HOLDS its headers, a // pre-header self-cut books zero, i.e. an under-count. On DeepSeek nothing is lost at all — its 200 // arrives on acceptance, so every self-cut there is post-header. Billable bool // Deliveries is how many times THIS request reached the provider inside one retry chain — counted by // DELIVERY, not by cause, so a chain that was cut once and answered an undecodable 2xx once reports // two. It is almost always 1; a broken connection is worth one retry, and then the provider has been // asked to generate TWICE while the ledger books ONE estimate — the store cannot write two settles // under one key. The number is carried rather than acted on: making the gap visible is this pack's // business, deciding what it costs is the owner's. Deliveries int // Err is the transport error that ended the attempt. Err error // Parent is the parent context's own cause, set ONLY for CutByParent. It rides here so that // errors.Is(err, context.Canceled) keeps deciding the process exit code while errors.As still // finds this type: the exit contract and the money both need to be true of one error. Parent error } func (e *AttemptCutError) Error() string { where := "before any response byte" if e.AfterHeaders { where = fmt.Sprintf("after %d body bytes", e.BytesRead) if e.WhitespaceOnly { where += " (whitespace only)" } } return fmt.Sprintf("%s: delivered request cut by %s %s after %s: %v", e.Provider, e.Cause, where, e.Elapsed.Round(time.Millisecond), e.Err) } // Unwrap returns both truths of a cancelled attempt. Go's errors.Is/As walk every branch, so the // caller that asks «did the run end» and the caller that asks «what did it interrupt» both get a // straight answer from the same value. func (e *AttemptCutError) Unwrap() []error { if e.Parent != nil { return []error{e.Err, e.Parent} } return []error{e.Err} } // deliveryTrace records the two httptrace facts the money boundary is drawn from. The callbacks run // on the transport's goroutine while the caller may already be reading the fields (a deadline fires // concurrently with a write completing), so both are atomics rather than plain bools. type deliveryTrace struct { wrote atomic.Bool firstByte atomic.Bool } func (t *deliveryTrace) clientTrace() *httptrace.ClientTrace { return &httptrace.ClientTrace{ // info.Err non-nil means the write itself failed — the request did NOT reach the provider, // and treating that as delivery would settle money for a call nobody received. WroteRequest: func(info httptrace.WroteRequestInfo) { if info.Err == nil { t.wrote.Store(true) } }, GotFirstResponseByte: func() { t.firstByte.Store(true) }, } } // delivered reports that the request reached the provider. `answered` says the transport handed us an // actual 2xx response, and it is the second half of the evidence — needed, and needed CONDITIONALLY. // // ⛔ A RESPONSE BYTE IS DELIVERY EVIDENCE, BUT ONLY WHEN A REPLY ACTUALLY CAME BACK TO US. net/http does // not wait for the write loop before handing back a response: Request.write fires WroteRequest from a // deferred call on the writeLoop goroutine (net/http/request.go) while roundTrip returns as soon as the // response channel fires (net/http/transport.go), and golang.org/x/net/http2 has the same shape. So a // provider answering EARLY — precisely what DeepSeek documents doing while a request waits to be // scheduled — can have its 200 overtake our own «the request was written» callback whenever the body is // still draining. Read as «not delivered», that call books $0 and is retried: measured at three re-asks. // // ⛔⛔ AND THE OTHER READING COSTS MORE. Taking the response BYTE alone as proof, whether or not a reply // reached us, pays for refusals: a provider that answers 401/403/413 and resets the connection while our // body is still writing gives GotFirstResponseByte=true and a WRITE ERROR, so http.Client.Do returns the // write failure and we never see the status at all. Measured on a copy of this tree: 22 refusals in 25 // booked a paid cut, one of them $0.80 for a request the provider declined — and the delivered-cut retry // sent the whole body a second time. When Do fails we hold no status line and cannot tell a refusal from // an early success, so only the write counts; when Do SUCCEEDS with a 2xx, the reply is proof by itself. func (t *deliveryTrace) delivered(answered bool) bool { return t.wrote.Load() || (answered && t.firstByte.Load()) } func (t *deliveryTrace) afterHeaders() bool { return t.firstByte.Load() } // causeOf names who ended a delivered attempt. The order is the meaning: the PARENT is asked first, // because a run that is ending has cancelled the attempt context too, and reading our own deadline // first would file every stopped run as a self-cut and flag chunks nobody's provider misbehaved on. func causeOf(ctx, attemptCtx context.Context) CutCause { switch { case ctx.Err() != nil: return CutByParent case errors.Is(attemptCtx.Err(), context.DeadlineExceeded): return CutBySelfDeadline default: return CutByConnection } } // cutError builds the typed error for a delivered attempt, or NIL when the request never went out. // The «not delivered» path is the one case where nothing was bought, and it must stay exactly what it // was: a plain retryable transport failure that releases the reservation. Nil rather than «the error // unchanged» so the caller branches on a value instead of on error identity — the shape that survives // somebody wrapping the transport error one layer deeper. // `answered` is true only where a 2xx response object is in hand — see delivered. func (c *openAIClient) cutError(ctx, attemptCtx context.Context, tr *deliveryTrace, started time.Time, body []byte, err error, answered bool) *AttemptCutError { if !tr.delivered(answered) { return nil } cause := causeOf(ctx, attemptCtx) cut := &AttemptCutError{ Provider: c.name, Cause: cause, Delivered: true, AfterHeaders: tr.afterHeaders(), Billable: answered && tr.afterHeaders(), BytesRead: len(body), WhitespaceOnly: len(body) == 0 || strings.TrimSpace(string(body)) == "", Elapsed: time.Since(started), Err: err, } if cause == CutByParent { cut.Parent = ctx.Err() } return cut } // retryable is the retry half of the disposition, kept beside the causes so the two cannot drift. // Only a broken connection is worth another call: our own deadline firing means the provider is STILL // GENERATING what we just stopped listening to, and retrying buys that generation a second time — the // defect this file exists to remove. A cancelled run has nothing to retry into. func (e *AttemptCutError) retryable() bool { return e.Cause == CutByConnection } // --- the deadline itself (backlog row 360 point 5, row 369) --- // The industry reference for sizing a non-streaming call, and the source of the default rate: // anthropic-sdk-go's CalculateNonStreamingTimeout budgets 1h · max_tokens / 128 000 and REFUSES a // request whose expected time exceeds ten minutes («streaming is required»). Ours is the same // arithmetic with the rate made per-provider data and the vendor's queue wait added, because we have // no streaming path to fall back to and must wait instead of refusing (research/21 §Q6 + its 08.09 // errata; openai-go's ResponseHeaderTimeout is the same idea applied to time-to-headers alone). // // It is the default because it is the one number that is not ours to invent — and it is the number // `attempt_s: 240` was silently standing in for: 240 s at this rate is ~8.5k tokens, the DRAFT's // budget, and the same 240 was carried to an editor budgeted at 16 000 with a doubling to 32 000. const ( vendorHourlyTokenBudget = 128000 vendorBudgetWindow = time.Hour ) // defaultTokensPerSecFloor is DERIVED from the pair above and never typed as a decimal: writing // 35.5 here would be a second carrier of a number the vendor states as 128 000 per hour, and the two // would drift the day either moved. func defaultTokensPerSecFloor() float64 { return float64(vendorHourlyTokenBudget) / vendorBudgetWindow.Seconds() } // deriveDeadline is the UNCLAMPED time this profile says a call for maxTokens needs: the vendor's // documented wait before generation starts, plus the generation itself at the slowest speed the // model has been observed to hold. It is a pure function of the profile and the budget — the budget // the call actually carries, so an escalated attempt that doubled its max_tokens doubles its time // instead of inheriting the budget of an attempt that asked for half as much. // // An unset or nonsensical floor falls back to the vendor default rather than to a division by zero // or a zero deadline: a config that forgot the field must wait too long, never not at all. func (p RetryProfile) deriveDeadline(maxTokens int) time.Duration { floor := p.TokensPerSecFloor if floor <= 0 { floor = defaultTokensPerSecFloor() } if maxTokens < 0 { maxTokens = 0 } return p.QueueSlack + time.Duration(float64(maxTokens)/floor*float64(time.Second)) } // DeadlineFor clamps the derivation between the configured attempt_s and attempt_max_s. It is exported // because it answers a question about a CONFIGURATION rather than about a call in flight — «how long // will one call of this size wait under this profile» — and the catalogue gate over models.yaml has to // ask it of a provider nobody has called yet. // // ⛔ attempt_s is the FLOOR, not the value. That is what makes this change safe to land on every // existing config at once: a call whose derived time is shorter than the configured deadline keeps // the configured one, so no provider loses a second it has today, and only calls that provably // could not finish get more. attempt_max_s bounds the other end — an operator's ceiling on how long // one call may hold a reservation — and is inert when unset. func (p RetryProfile) DeadlineFor(maxTokens int) time.Duration { d := p.deriveDeadline(maxTokens) if d < p.AttemptTimeout { d = p.AttemptTimeout } if p.AttemptMax > 0 && d > p.AttemptMax { d = p.AttemptMax } return d } // attemptDeadline is the client's own wrapper: the same clamp, plus the ONE notice an operator needs // when a provider is running on the vendor default. Waiting a quarter of an hour for a call is a // legitimate configuration — the owner ratified waiting up to ~20 minutes — but it must never be a // surprise, and «why is nothing happening» is the pain this line answers before it is felt. // // The condition is structural rather than a threshold in tokens: it fires exactly when the derived // time OVERRIDES the configured attempt_s, i.e. when this call is one the configured deadline could // not have covered. A provider whose calls all fit inside its own attempt_s never sees it. func (c *openAIClient) attemptDeadline(ctx context.Context, maxTokens int) time.Duration { d := c.profile.DeadlineFor(maxTokens) if c.profile.TokensPerSecFloor <= 0 && d > c.profile.AttemptTimeout { c.floorWarned.Do(func() { if c.log == nil { return } c.log.WarnContext(ctx, "no measured generation speed for this provider; the call deadline is derived from the vendor default and is longer than the configured attempt_s — set timeouts.tok_s_floor from this provider's own request_log after the first run", "provider", c.name, "default_tok_s", fmt.Sprintf("%.1f", defaultTokensPerSecFloor()), "max_tokens", maxTokens, "derived_deadline", d.String(), "attempt_s", c.profile.AttemptTimeout.String()) }) } return d } // cancelledDuring is what a stopped run returns: BOTH the cancellation, which decides the process exit // code, and whatever the attempt underneath it was — which is what decides the money. // // ⛔ IT IS errors.Join AND NOT A CHOICE BETWEEN THEM. Returning the attempt's error alone loses // `errors.Is(err, context.Canceled)` for every attempt error that does not happen to wrap the // cancellation — a stopped run would then leave with a foreign exit code. Returning the cancellation // alone loses `errors.As`, and with it the money for a call already on the wire, which is the defect // this whole file exists to remove. Join keeps both true of one value, and it does so whatever the // attempt's cause was: an earlier form of this guard preserved only the errors that already carried the // parent's own error, so a connection_lost or attempt_timeout whose run was stopped a moment later was // still silently reduced to a bare cancellation. func cancelledDuring(ctxErr, attemptErr error) error { if attemptErr == nil { return ctxErr } if errors.Is(attemptErr, ctxErr) { return attemptErr // it already carries both; joining would only duplicate the sentence } return errors.Join(ctxErr, attemptErr) }