textmachine/backend/internal/llm/attemptcut.go

package llm

import (
	"context"
	"errors"
	"fmt"
	"net/http/httptrace"
	"strings"
	"sync/atomic"
	"time"
)

// attemptcut.go: what OUR OWN deadline does to a call, and the deadline itself.
//
// A provider that has accepted a request generates whether or not we are still listening. When the
// attempt deadline fires mid-generation the connection drops on our side, the provider never learns
// it, and the call is billed all the same — so «we stopped waiting» is a MONEY event, not a transport
// one. The transport used to lose it twice over: the body read's error went to `_`, and truncation was
// judged by SIZE, so a body our own deadline cut short (small) was indistinguishable from a whole one
// and went down the retryable «broken connection» branch that the same code's comment forbids for a
// billed 2xx («each retry is a new billed 2xx»).
//
// THE ONE FACT COMMON TO EVERY SUCH CASE IS THAT THE REQUEST WAS DELIVERED, and it is observable
// without a byte on the wire: net/http/httptrace reports when the request has been WRITTEN and when
// the first response byte arrived.
//
// ⚠ NO SURVEYED HARNESS DRAWS THE BOUNDARY HERE, and that is a fact rather than a claim of novelty:
// research/21 covers eleven of them and mentions httptrace nowhere (0 hits against 20 for openai-go).
// The reason is what they are: five are stream-only and four do both (§Q6), so «did anything arrive» is
// answered by the first event and the question never comes up; the remaining two are generic SDKs whose
// non-streaming answer is to REFUSE a long call outright. On a non-streaming path with no refusal to
// fall back on it has to be answered another way, and the stdlib already answers it — so the mechanism
// is stdlib rather than ours (research/21's own rule: an industrial primary source before a home-made one).
//
// Those two booleans separate the three cases that a status line cannot:
//
//	delivered, headers still pending, our deadline   WroteRequest=true  GotFirstResponseByte=false
//	TLS handshake hanging — NOT delivered            WroteRequest=false GotFirstResponseByte=false
//	early 200 with empty lines, our deadline         WroteRequest=true  GotFirstResponseByte=true
//
// The middle row is the ONLY one where «nothing was bought» is true by construction. On DeepSeek the
// first and third are the live ones: the vendor documents an early 200 with empty lines while a
// request waits to be scheduled, so a 200 there means ACCEPTED, not GENERATED, and the branch that
// reads «No 2xx ever arrived: nothing was billed» is unreachable on it — every self-cut hid under
// decode_error instead.

// CutCause says WHO ended an attempt that had already been delivered. The three are dispositions,
// not shades of one error: our own deadline is not retried and is flagged, a run the operator
// stopped is re-done on resume at the same budget, and a broken connection gets one retry.
type CutCause string

const (
	// CutBySelfDeadline: our per-attempt deadline fired while the provider was still working.
	// Retrying buys the same generation a second time — the whole point of typing this.
	CutBySelfDeadline CutCause = "attempt_timeout"
	// CutByParent: the RUN ended (stop, Ctrl-C) under a call that had already gone out. The call was
	// healthy; a human stopped it.
	CutByParent CutCause = "cancelled"
	// CutByConnection: the connection broke after delivery (unexpected EOF, an h2 stream reset) with
	// both deadlines still alive. Transport-shaped, so it is worth exactly one retry.
	CutByConnection CutCause = "connection_lost"
)

// AttemptCutError is a DELIVERED request whose reply we did not receive whole.
//
// ⚠ It is deliberately NOT named for the timeout: two of its three causes are not one (a run somebody
// stopped, a connection that broke), and the runner routes money, flag and resume differently for each.
// A type named `AttemptTimeoutError` carrying `Cause: cancelled` would be a name lying about a cause,
// which is the thing the flag vocabulary forbids outright (D39.93 п.2).
//
// Delivered is the money boundary and is always true on a value that reaches a caller — the
// constructor refuses to build one otherwise, because «not delivered» is the case that must keep
// costing nothing.
type AttemptCutError struct {
	Provider string
	Cause    CutCause
	// Delivered: the request bytes reached the provider (httptrace WroteRequest, no write error).
	Delivered bool
	// AfterHeaders: the first BYTE of a response had arrived when the call was cut (httptrace's
	// GotFirstResponseByte) — the early-200 case.
	//
	// ⚠ IT IS A BYTE, NOT A REPLY, and the distinction is the whole reason `Billable` is not this field.
	// The first byte fires for a 1xx, for a header block that never terminated, for a broken proxy's
	// garbage — none of which is a provider acknowledging anything. Read as «a response arrived» it
	// tells an operator about a reply that did not exist; read as what it is, it is the evidence that
	// something came back down the socket, and the money question is answered by `Billable`, which also
	// requires a 2xx object in hand.
	AfterHeaders bool
	// BytesRead is how much of the body we did get. It is EVIDENCE, never the criterion: judging
	// truncation by size is the defect this type replaces.
	BytesRead int
	// WhitespaceOnly says the bytes we got carry no content at all (DeepSeek's documented empty lines
	// while a request waits for scheduling). It distinguishes «the provider was still queueing» from
	// «the provider was mid-answer», which is the difference between a wait worth extending and a
	// generation worth not buying twice.
	WhitespaceOnly bool
	Elapsed        time.Duration
	// Billable is the MONEY boundary, and it is deliberately NARROWER than Delivered.
	//
	// ⛔ TWO PREDICATES, BECAUSE THEY ANSWER TWO QUESTIONS. `WroteRequest` says our bytes left for the
	// peer's TCP window; it says nothing about the APPLICATION behind it — a load balancer can accept
	// while the backend never sees the request. Measured on a stopped run: 3 of 25 cancelled in-flight
	// calls settled an estimate for a request no handler ever entered. On a non-streaming wire the one
	// signal that a provider's application acknowledged the request is a 2xx response object reaching
	// US, so that is what money is drawn on. Delivery still drives RETRY — a written request is not
	// automatically re-sent — and that boundary is unchanged.
	//
	// The direction of what remains is the tolerable one: on a provider that HOLDS its headers, a
	// pre-header self-cut books zero, i.e. an under-count. On DeepSeek nothing is lost at all — its 200
	// arrives on acceptance, so every self-cut there is post-header.
	Billable bool
	// Deliveries is how many times THIS request reached the provider inside one retry chain — counted by
	// DELIVERY, not by cause, so a chain that was cut once and answered an undecodable 2xx once reports
	// two. It is almost always 1; a broken connection is worth one retry, and then the provider has been
	// asked to generate TWICE while the ledger books ONE estimate — the store cannot write two settles
	// under one key. The number is carried rather than acted on: making the gap visible is this pack's
	// business, deciding what it costs is the owner's.
	Deliveries int
	// Err is the transport error that ended the attempt.
	Err error
	// Parent is the parent context's own cause, set ONLY for CutByParent. It rides here so that
	// errors.Is(err, context.Canceled) keeps deciding the process exit code while errors.As still
	// finds this type: the exit contract and the money both need to be true of one error.
	Parent error
}

func (e *AttemptCutError) Error() string {
	where := "before any response byte"
	if e.AfterHeaders {
		where = fmt.Sprintf("after %d body bytes", e.BytesRead)
		if e.WhitespaceOnly {
			where += " (whitespace only)"
		}
	}
	return fmt.Sprintf("%s: delivered request cut by %s %s after %s: %v",
		e.Provider, e.Cause, where, e.Elapsed.Round(time.Millisecond), e.Err)
}

// Unwrap returns both truths of a cancelled attempt. Go's errors.Is/As walk every branch, so the
// caller that asks «did the run end» and the caller that asks «what did it interrupt» both get a
// straight answer from the same value.
func (e *AttemptCutError) Unwrap() []error {
	if e.Parent != nil {
		return []error{e.Err, e.Parent}
	}
	return []error{e.Err}
}

// deliveryTrace records the two httptrace facts the money boundary is drawn from. The callbacks run
// on the transport's goroutine while the caller may already be reading the fields (a deadline fires
// concurrently with a write completing), so both are atomics rather than plain bools.
type deliveryTrace struct {
	wrote     atomic.Bool
	firstByte atomic.Bool
}

func (t *deliveryTrace) clientTrace() *httptrace.ClientTrace {
	return &httptrace.ClientTrace{
		// info.Err non-nil means the write itself failed — the request did NOT reach the provider,
		// and treating that as delivery would settle money for a call nobody received.
		WroteRequest: func(info httptrace.WroteRequestInfo) {
			if info.Err == nil {
				t.wrote.Store(true)
			}
		},
		GotFirstResponseByte: func() { t.firstByte.Store(true) },
	}
}

// delivered reports that the request reached the provider. `answered` says the transport handed us an
// actual 2xx response, and it is the second half of the evidence — needed, and needed CONDITIONALLY.
//
// ⛔ A RESPONSE BYTE IS DELIVERY EVIDENCE, BUT ONLY WHEN A REPLY ACTUALLY CAME BACK TO US. net/http does
// not wait for the write loop before handing back a response: Request.write fires WroteRequest from a
// deferred call on the writeLoop goroutine (net/http/request.go) while roundTrip returns as soon as the
// response channel fires (net/http/transport.go), and golang.org/x/net/http2 has the same shape. So a
// provider answering EARLY — precisely what DeepSeek documents doing while a request waits to be
// scheduled — can have its 200 overtake our own «the request was written» callback whenever the body is
// still draining. Read as «not delivered», that call books $0 and is retried: measured at three re-asks.
//
// ⛔⛔ AND THE OTHER READING COSTS MORE. Taking the response BYTE alone as proof, whether or not a reply
// reached us, pays for refusals: a provider that answers 401/403/413 and resets the connection while our
// body is still writing gives GotFirstResponseByte=true and a WRITE ERROR, so http.Client.Do returns the
// write failure and we never see the status at all. Measured on a copy of this tree: 22 refusals in 25
// booked a paid cut, one of them $0.80 for a request the provider declined — and the delivered-cut retry
// sent the whole body a second time. When Do fails we hold no status line and cannot tell a refusal from
// an early success, so only the write counts; when Do SUCCEEDS with a 2xx, the reply is proof by itself.
func (t *deliveryTrace) delivered(answered bool) bool {
	return t.wrote.Load() || (answered && t.firstByte.Load())
}
func (t *deliveryTrace) afterHeaders() bool { return t.firstByte.Load() }

// causeOf names who ended a delivered attempt. The order is the meaning: the PARENT is asked first,
// because a run that is ending has cancelled the attempt context too, and reading our own deadline
// first would file every stopped run as a self-cut and flag chunks nobody's provider misbehaved on.
func causeOf(ctx, attemptCtx context.Context) CutCause {
	switch {
	case ctx.Err() != nil:
		return CutByParent
	case errors.Is(attemptCtx.Err(), context.DeadlineExceeded):
		return CutBySelfDeadline
	default:
		return CutByConnection
	}
}

// cutError builds the typed error for a delivered attempt, or NIL when the request never went out.
// The «not delivered» path is the one case where nothing was bought, and it must stay exactly what it
// was: a plain retryable transport failure that releases the reservation. Nil rather than «the error
// unchanged» so the caller branches on a value instead of on error identity — the shape that survives
// somebody wrapping the transport error one layer deeper.
// `answered` is true only where a 2xx response object is in hand — see delivered.
func (c *openAIClient) cutError(ctx, attemptCtx context.Context, tr *deliveryTrace, started time.Time, body []byte, err error, answered bool) *AttemptCutError {
	if !tr.delivered(answered) {
		return nil
	}
	cause := causeOf(ctx, attemptCtx)
	cut := &AttemptCutError{
		Provider: c.name, Cause: cause,
		Delivered: true, AfterHeaders: tr.afterHeaders(),
		Billable:  answered && tr.afterHeaders(),
		BytesRead: len(body), WhitespaceOnly: len(body) == 0 || strings.TrimSpace(string(body)) == "",
		Elapsed: time.Since(started), Err: err,
	}
	if cause == CutByParent {
		cut.Parent = ctx.Err()
	}
	return cut
}

// retryable is the retry half of the disposition, kept beside the causes so the two cannot drift.
// Only a broken connection is worth another call: our own deadline firing means the provider is STILL
// GENERATING what we just stopped listening to, and retrying buys that generation a second time — the
// defect this file exists to remove. A cancelled run has nothing to retry into.
func (e *AttemptCutError) retryable() bool { return e.Cause == CutByConnection }

// --- the deadline itself (backlog row 360 point 5, row 369) ---

// The industry reference for sizing a non-streaming call, and the source of the default rate:
// anthropic-sdk-go's CalculateNonStreamingTimeout budgets 1h · max_tokens / 128 000 and REFUSES a
// request whose expected time exceeds ten minutes («streaming is required»). Ours is the same
// arithmetic with the rate made per-provider data and the vendor's queue wait added, because we have
// no streaming path to fall back to and must wait instead of refusing (research/21 §Q6 + its 08.09
// errata; openai-go's ResponseHeaderTimeout is the same idea applied to time-to-headers alone).
//
// It is the default because it is the one number that is not ours to invent — and it is the number
// `attempt_s: 240` was silently standing in for: 240 s at this rate is ~8.5k tokens, the DRAFT's
// budget, and the same 240 was carried to an editor budgeted at 16 000 with a doubling to 32 000.
const (
	vendorHourlyTokenBudget = 128000
	vendorBudgetWindow      = time.Hour
)

// defaultTokensPerSecFloor is DERIVED from the pair above and never typed as a decimal: writing
// 35.5 here would be a second carrier of a number the vendor states as 128 000 per hour, and the two
// would drift the day either moved.
func defaultTokensPerSecFloor() float64 {
	return float64(vendorHourlyTokenBudget) / vendorBudgetWindow.Seconds()
}

// deriveDeadline is the UNCLAMPED time this profile says a call for maxTokens needs: the vendor's
// documented wait before generation starts, plus the generation itself at the slowest speed the
// model has been observed to hold. It is a pure function of the profile and the budget — the budget
// the call actually carries, so an escalated attempt that doubled its max_tokens doubles its time
// instead of inheriting the budget of an attempt that asked for half as much.
//
// An unset or nonsensical floor falls back to the vendor default rather than to a division by zero
// or a zero deadline: a config that forgot the field must wait too long, never not at all.
func (p RetryProfile) deriveDeadline(maxTokens int) time.Duration {
	floor := p.TokensPerSecFloor
	if floor <= 0 {
		floor = defaultTokensPerSecFloor()
	}
	if maxTokens < 0 {
		maxTokens = 0
	}
	return p.QueueSlack + time.Duration(float64(maxTokens)/floor*float64(time.Second))
}

// DeadlineFor clamps the derivation between the configured attempt_s and attempt_max_s. It is exported
// because it answers a question about a CONFIGURATION rather than about a call in flight — «how long
// will one call of this size wait under this profile» — and the catalogue gate over models.yaml has to
// ask it of a provider nobody has called yet.
//
// ⛔ attempt_s is the FLOOR, not the value. That is what makes this change safe to land on every
// existing config at once: a call whose derived time is shorter than the configured deadline keeps
// the configured one, so no provider loses a second it has today, and only calls that provably
// could not finish get more. attempt_max_s bounds the other end — an operator's ceiling on how long
// one call may hold a reservation — and is inert when unset.
func (p RetryProfile) DeadlineFor(maxTokens int) time.Duration {
	d := p.deriveDeadline(maxTokens)
	if d < p.AttemptTimeout {
		d = p.AttemptTimeout
	}
	if p.AttemptMax > 0 && d > p.AttemptMax {
		d = p.AttemptMax
	}
	return d
}

// attemptDeadline is the client's own wrapper: the same clamp, plus the ONE notice an operator needs
// when a provider is running on the vendor default. Waiting a quarter of an hour for a call is a
// legitimate configuration — the owner ratified waiting up to ~20 minutes — but it must never be a
// surprise, and «why is nothing happening» is the pain this line answers before it is felt.
//
// The condition is structural rather than a threshold in tokens: it fires exactly when the derived
// time OVERRIDES the configured attempt_s, i.e. when this call is one the configured deadline could
// not have covered. A provider whose calls all fit inside its own attempt_s never sees it.
func (c *openAIClient) attemptDeadline(ctx context.Context, maxTokens int) time.Duration {
	d := c.profile.DeadlineFor(maxTokens)
	if c.profile.TokensPerSecFloor <= 0 && d > c.profile.AttemptTimeout {
		c.floorWarned.Do(func() {
			if c.log == nil {
				return
			}
			c.log.WarnContext(ctx, "no measured generation speed for this provider; the call deadline is derived from the vendor default and is longer than the configured attempt_s — set timeouts.tok_s_floor from this provider's own request_log after the first run",
				"provider", c.name, "default_tok_s", fmt.Sprintf("%.1f", defaultTokensPerSecFloor()),
				"max_tokens", maxTokens, "derived_deadline", d.String(), "attempt_s", c.profile.AttemptTimeout.String())
		})
	}
	return d
}

// cancelledDuring is what a stopped run returns: BOTH the cancellation, which decides the process exit
// code, and whatever the attempt underneath it was — which is what decides the money.
//
// ⛔ IT IS errors.Join AND NOT A CHOICE BETWEEN THEM. Returning the attempt's error alone loses
// `errors.Is(err, context.Canceled)` for every attempt error that does not happen to wrap the
// cancellation — a stopped run would then leave with a foreign exit code. Returning the cancellation
// alone loses `errors.As`, and with it the money for a call already on the wire, which is the defect
// this whole file exists to remove. Join keeps both true of one value, and it does so whatever the
// attempt's cause was: an earlier form of this guard preserved only the errors that already carried the
// parent's own error, so a connection_lost or attempt_timeout whose run was stopped a moment later was
// still silently reduced to a bare cancellation.
func cancelledDuring(ctxErr, attemptErr error) error {
	if attemptErr == nil {
		return ctxErr
	}
	if errors.Is(attemptErr, ctxErr) {
		return attemptErr // it already carries both; joining would only duplicate the sentence
	}
	return errors.Join(ctxErr, attemptErr)
}