feat(ai-bot): local LLM voice backend — OpenAI-compatible adapter, health-probed failover to Grok, billing by actual serving model ($0 local), LOCAL_LLM_* config
This commit is contained in:
parent
ddc5cd2cad
commit
0a538a9e0b
17 changed files with 1129 additions and 21 deletions
|
|
@ -186,6 +186,36 @@ offline-eval gate (misroute < 2-3% AND measured saving > the second provider's c
|
|||
| `GEMINI_MODEL` | `gemini-2.5-flash-lite` | cheap model for trivial/classifier |
|
||||
| `GEMINI_BASE_URL` | `…/v1beta/openai` | OpenAI-compat endpoint (native grounding endpoint derived from it) |
|
||||
|
||||
### Local LLM backend (voice failover, **default OFF**)
|
||||
|
||||
Phase 1 of `docs/plans/local_llm_backend.md`: while the self-hosted backend is reachable, the
|
||||
bot's **voice** (every `b.llm` call — grok_direct, reasoning, web/project synthesis) is served by
|
||||
it **for free**; when it isn't, requests transparently fall back to cloud Grok ([failover.go](failover.go),
|
||||
active `GET <base>/models` probe + circuit breaker). Works with **any OpenAI-compatible server**
|
||||
(ollama, llama.cpp `llama-server`, vLLM, LM Studio) — the adapter ([provider_local.go](provider_local.go))
|
||||
speaks only the standard `/chat/completions` + `/models` surface, selected purely by env. The
|
||||
classifier (`b.gemini`) and web fetch (`b.web`) stay cloud until their own phases. A local answer
|
||||
books **$0** in the ledger (billing follows `LLMResponse.Model` — the model that actually served);
|
||||
`request_log.models.final` records which backend answered, so the local-vs-cloud share and quality
|
||||
(via reaction feedback) are measurable. `XAI_API_KEY` stays **required** — the cloud is the fallback leg.
|
||||
|
||||
| Env | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `LOCAL_LLM_ENABLED` | false | wrap `b.llm` in the local-first failover (off = byte-identical to today) |
|
||||
| `LOCAL_LLM_BASE_URL` | — | OpenAI-compatible `/v1` root, e.g. `http://172.18.0.1:11434/v1` (the SSH-tunnel relay on the docker bridge). **Required when enabled** (fail-fast). |
|
||||
| `LOCAL_LLM_MODEL` | `huihui_ai/qwen3-abliterated:8b` | the local server's model tag. Must **differ** from the cloud model names (its $0 price entry would zero cloud billing — refuses to boot). |
|
||||
| `LOCAL_LLM_API_KEY` / `_FILE` | — | optional; empty sends **no** Authorization header |
|
||||
| `LOCAL_LLM_TEMP` | 0.7 | temperature the local adapter sends (Qwen3 non-thinking tuning), **overriding** `XAI_TEMPERATURE` — a request temp beats the Modelfile, so passing Grok's 0.6 through would mistune the local model. The other Qwen3 samplers (top_k/min_p/repeat_penalty) can't cross ollama's `/v1` and live in the Modelfile at home. |
|
||||
| `LOCAL_LLM_MAX_OUTPUT_TOKENS` | 1024 | replaces `MAX_OUTPUT_TOKENS` on the local leg (0 = inherit). ollama's cap covers **thinking + answer together** (xAI bills thinking on top), so a thinking route (`reasoning_effort` low/high — the web/project synthesis and reason routes) on the cloud-sized 320 would burn the whole budget on reasoning and return empty content. Local tokens are free. |
|
||||
| `LOCAL_LLM_TIMEOUT_SECONDS` | 45 | the local leg's own request deadline — a half-alive tunnel fails fast enough to leave `REQUEST_BUDGET_SECONDS` room for the cloud fallback |
|
||||
| `LOCAL_LLM_HEALTH_INTERVAL_SECONDS` | 15 | background health-probe period. Recovery is probe-driven only (anti-flapping); a request-path trip needs 2 consecutive probe passes to re-close. |
|
||||
|
||||
`reasoning_effort` passes through unchanged: ollama's `/v1` maps it onto Qwen3 thinking (`none` =
|
||||
off — the casual `GROK_REASONING_EFFORT_DIRECT=none` route; `low`/`high` = on; verified live), so
|
||||
the cascade's existing per-route efforts drive local thinking with **no new knob**. A local
|
||||
**terminal 4xx** (wrong model tag) fails loud instead of falling back — a config error must not be
|
||||
masked by paid cloud answers; 429/5xx/timeouts/transport errors fall back within the same request.
|
||||
|
||||
## One-time setup (appservice registration)
|
||||
|
||||
Like the mautrix bridges (e.g. telegram), the bot **generates its own
|
||||
|
|
|
|||
|
|
@ -80,6 +80,14 @@ type Bot struct {
|
|||
func NewBot(ctx context.Context, cfg *Config, logger *slog.Logger) (*Bot, error) {
|
||||
mx := NewMatrixClient(cfg.HomeserverURL, cfg.ASToken, cfg.BotMXID)
|
||||
llm := NewXAIClient(cfg.XAIBaseURL, cfg.XAIAPIKey, logger)
|
||||
// Local-first voice (Phase 1, docs/plans/local_llm_backend.md): wrap the cloud
|
||||
// client in the failover decorator — the home backend answers while its tunnel is
|
||||
// healthy, Grok otherwise. Only b.llm changes; the classifier (b.gemini) and web
|
||||
// layer (b.web) stay cloud until their own phases. The prober lives on ctx (the
|
||||
// process signal context), same lifetime as the bot.
|
||||
if cfg.LocalLLMEnabled {
|
||||
llm = NewFailoverLLMClient(ctx, NewLocalLLMClient(cfg, logger), llm, cfg, logger)
|
||||
}
|
||||
|
||||
st, err := OpenStore(cfg.DatabaseURL)
|
||||
if err != nil {
|
||||
|
|
|
|||
|
|
@ -209,12 +209,26 @@ func (b *Bot) genGrokDirect(ctx context.Context, msgs []Message, convID string,
|
|||
if err != nil {
|
||||
return err
|
||||
}
|
||||
res.route, res.finalModel = routeGrokDirect, b.cfg.XAIModel
|
||||
model := servedModel(b.cfg.XAIModel, resp)
|
||||
res.route, res.finalModel = routeGrokDirect, model
|
||||
res.text, res.usage, res.providerID = resp.Text, resp.Usage, resp.ProviderRequestID
|
||||
res.cost.Token += computeUSD(b.cfg.XAIModel, resp.Usage, b.cfg)
|
||||
res.cost.Token += computeUSD(model, resp.Usage, b.cfg)
|
||||
return nil
|
||||
}
|
||||
|
||||
// servedModel is the model a completion must be billed and telemetered as: the one
|
||||
// that ACTUALLY answered when the adapter reports it (resp.Model — the local/cloud
|
||||
// failover can answer with a different backend than the cascade requested, and the
|
||||
// local one is priced $0), else the requested model. Without this, a free local
|
||||
// answer books at the requested Grok price and the ledger/ceiling lie (§5.4 of the
|
||||
// local-backend plan).
|
||||
func servedModel(requested string, resp *LLMResponse) string {
|
||||
if resp.Model != "" {
|
||||
return resp.Model
|
||||
}
|
||||
return requested
|
||||
}
|
||||
|
||||
// genTrivial answers a trivial message with the cheap Gemini model. An empty reply is
|
||||
// treated as a failure so the caller degrades to Grok rather than sending nothing.
|
||||
func (b *Bot) genTrivial(ctx context.Context, msgs []Message, res *genResult) error {
|
||||
|
|
@ -232,9 +246,10 @@ func (b *Bot) genTrivial(ctx context.Context, msgs []Message, res *genResult) er
|
|||
if strings.TrimSpace(resp.Text) == "" {
|
||||
return fmt.Errorf("trivial: empty Gemini reply")
|
||||
}
|
||||
res.route, res.finalModel = routeTrivial, b.cfg.GeminiModel
|
||||
model := servedModel(b.cfg.GeminiModel, resp)
|
||||
res.route, res.finalModel = routeTrivial, model
|
||||
res.text, res.usage, res.providerID = resp.Text, resp.Usage, resp.ProviderRequestID
|
||||
res.cost.Token += computeUSD(b.cfg.GeminiModel, resp.Usage, b.cfg)
|
||||
res.cost.Token += computeUSD(model, resp.Usage, b.cfg)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
@ -257,9 +272,10 @@ func (b *Bot) genReason(ctx context.Context, msgs []Message, convID string, res
|
|||
if strings.TrimSpace(resp.Text) == "" {
|
||||
return fmt.Errorf("reason: empty reply")
|
||||
}
|
||||
res.route, res.finalModel = routeReason, b.cfg.ReasoningModel
|
||||
model := servedModel(b.cfg.ReasoningModel, resp)
|
||||
res.route, res.finalModel = routeReason, model
|
||||
res.text, res.usage, res.providerID = resp.Text, resp.Usage, resp.ProviderRequestID
|
||||
res.cost.Token += computeUSD(b.cfg.ReasoningModel, resp.Usage, b.cfg)
|
||||
res.cost.Token += computeUSD(model, resp.Usage, b.cfg)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
@ -288,9 +304,10 @@ func (b *Bot) genProjectThenGrok(ctx context.Context, msgs []Message, convID str
|
|||
if strings.TrimSpace(resp.Text) == "" {
|
||||
return fmt.Errorf("project: empty reply")
|
||||
}
|
||||
res.route, res.finalModel = routeProject, b.cfg.XAIModel
|
||||
model := servedModel(b.cfg.XAIModel, resp)
|
||||
res.route, res.finalModel = routeProject, model
|
||||
res.text, res.usage, res.providerID = resp.Text, resp.Usage, resp.ProviderRequestID
|
||||
res.cost.Token += computeUSD(b.cfg.XAIModel, resp.Usage, b.cfg)
|
||||
res.cost.Token += computeUSD(model, resp.Usage, b.cfg)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
@ -370,7 +387,8 @@ func (b *Bot) genWebThenGrok(ctx context.Context, body string, isDM bool, msgs [
|
|||
if strings.TrimSpace(resp.Text) == "" {
|
||||
return fmt.Errorf("web synth: empty reply")
|
||||
}
|
||||
res.route, res.finalModel = routeWebThenGrok, b.cfg.XAIModel
|
||||
model := servedModel(b.cfg.XAIModel, resp)
|
||||
res.route, res.finalModel = routeWebThenGrok, model
|
||||
res.text, res.providerID = resp.Text, resp.ProviderRequestID
|
||||
// Report BOTH calls' tokens so the analytics token totals match the two-call route.
|
||||
res.usage = Usage{
|
||||
|
|
@ -379,7 +397,7 @@ func (b *Bot) genWebThenGrok(ctx context.Context, body string, isDM bool, msgs [
|
|||
CompletionTokens: resp.Usage.CompletionTokens + webUsage.CompletionTokens,
|
||||
ReasoningTokens: resp.Usage.ReasoningTokens + webUsage.ReasoningTokens,
|
||||
}
|
||||
res.cost.Token += computeUSD(b.cfg.XAIModel, resp.Usage, b.cfg)
|
||||
res.cost.Token += computeUSD(model, resp.Usage, b.cfg)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -11,10 +11,13 @@ import (
|
|||
|
||||
func discardLog() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
|
||||
|
||||
// fakeLLM is a scriptable LLMClient for dispatch/degrade tests.
|
||||
// fakeLLM is a scriptable LLMClient for dispatch/degrade tests. model, when set, is
|
||||
// reported as LLMResponse.Model (the responder identity a failover leg would fill);
|
||||
// empty keeps the pre-Model behaviour (billing falls back to the requested model).
|
||||
type fakeLLM struct {
|
||||
text string
|
||||
usage Usage
|
||||
model string
|
||||
err error
|
||||
calls int
|
||||
lastReq LLMRequest
|
||||
|
|
@ -26,7 +29,7 @@ func (f *fakeLLM) Complete(_ context.Context, req LLMRequest) (*LLMResponse, err
|
|||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
return &LLMResponse{Text: f.text, Usage: f.usage, ProviderRequestID: "fake"}, nil
|
||||
return &LLMResponse{Text: f.text, Usage: f.usage, Model: f.model, ProviderRequestID: "fake"}, nil
|
||||
}
|
||||
|
||||
type fakeWeb struct {
|
||||
|
|
@ -782,6 +785,75 @@ func TestReserveEstimateProjectNoBump(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestServedModel: billing follows the responder when the adapter reports it, else
|
||||
// the requested model (fakes / adapters predating LLMResponse.Model).
|
||||
func TestServedModel(t *testing.T) {
|
||||
if got := servedModel("grok-x", &LLMResponse{}); got != "grok-x" {
|
||||
t.Fatalf("empty resp.Model → %q, want the requested model", got)
|
||||
}
|
||||
if got := servedModel("grok-x", &LLMResponse{Model: "qwen-local"}); got != "qwen-local" {
|
||||
t.Fatalf("resp.Model set → %q, want the responder", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGenerateBillsByServedModel is the §5.4 money invariant: when the failover seam
|
||||
// answers with the free local model, the cascade books $0 (its price entry) and the
|
||||
// telemetry final model is the responder — NOT the requested Grok priced at Grok
|
||||
// rates. And the fallback leg answering keeps booking real cloud money.
|
||||
func TestGenerateBillsByServedModel(t *testing.T) {
|
||||
usage := Usage{PromptTokens: 1000, CompletionTokens: 500}
|
||||
|
||||
local := &fakeLLM{text: "local answer", usage: usage, model: "qwen-local"}
|
||||
cfg := cascadeCfg()
|
||||
cfg.Prices["qwen-local"] = ModelPrice{} // the $0 entry LoadConfig registers
|
||||
b := &Bot{cfg: &cfg, llm: local, log: discardLog()}
|
||||
res, err := b.generate(context.Background(), "привет", msgs("привет"), "", true)
|
||||
if err != nil {
|
||||
t.Fatalf("generate: %v", err)
|
||||
}
|
||||
if res.cost.Token != 0 {
|
||||
t.Fatalf("local answer booked $%v, want $0 (the easiest-to-miss bug)", res.cost.Token)
|
||||
}
|
||||
if res.finalModel != "qwen-local" {
|
||||
t.Fatalf("finalModel = %q, want the responder qwen-local (telemetry backend attribution)", res.finalModel)
|
||||
}
|
||||
|
||||
cloud := &fakeLLM{text: "cloud answer", usage: usage, model: "grok-x"}
|
||||
cfg2 := cascadeCfg()
|
||||
b2 := &Bot{cfg: &cfg2, llm: cloud, log: discardLog()}
|
||||
res2, err := b2.generate(context.Background(), "привет", msgs("привет"), "", true)
|
||||
if err != nil {
|
||||
t.Fatalf("generate: %v", err)
|
||||
}
|
||||
if res2.cost.Token <= 0 {
|
||||
t.Fatalf("cloud answer booked $%v, want real Grok money", res2.cost.Token)
|
||||
}
|
||||
if res2.finalModel != "grok-x" {
|
||||
t.Fatalf("finalModel = %q, want grok-x", res2.finalModel)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGenerateReasonRouteBillsByServedModel: the $0 invariant holds on the reasoning
|
||||
// route too (a different computeUSD call-site — the audit found four).
|
||||
func TestGenerateReasonRouteBillsByServedModel(t *testing.T) {
|
||||
local := &fakeLLM{text: "deep local answer", usage: Usage{PromptTokens: 100, CompletionTokens: 400}, model: "qwen-local"}
|
||||
cfg := cascadeCfg()
|
||||
cfg.ReasoningEnabled = true
|
||||
cfg.Prices["qwen-local"] = ModelPrice{}
|
||||
b := &Bot{cfg: &cfg, llm: local, log: discardLog()}
|
||||
|
||||
res, err := b.generate(context.Background(), "подумай глубже про X", msgs("подумай глубже про X"), "", true)
|
||||
if err != nil {
|
||||
t.Fatalf("generate: %v", err)
|
||||
}
|
||||
if res.route != routeReason {
|
||||
t.Fatalf("route = %q, want reason", res.route)
|
||||
}
|
||||
if res.cost.Token != 0 || res.finalModel != "qwen-local" {
|
||||
t.Fatalf("cost=%v finalModel=%q, want $0/qwen-local", res.cost.Token, res.finalModel)
|
||||
}
|
||||
}
|
||||
|
||||
func hedgeContains(ms []Message, sub string) bool {
|
||||
for _, m := range ms {
|
||||
if strings.Contains(m.Content, sub) {
|
||||
|
|
|
|||
|
|
@ -157,6 +157,33 @@ type Config struct {
|
|||
GeminiAPIKey string
|
||||
GeminiModel string
|
||||
|
||||
// --- Local LLM backend (Phase 1 of docs/plans/local_llm_backend.md): the home
|
||||
// inference server behind the SSH tunnel serves the bot's voice for FREE while
|
||||
// reachable; cloud Grok stays REQUIRED as the transparent fallback (failover.go).
|
||||
// Off (default) → b.llm is the plain xAI client, byte-identical to today. ---
|
||||
|
||||
LocalLLMEnabled bool
|
||||
LocalLLMBaseURL string // OpenAI-compatible /v1 root, e.g. http://172.18.0.1:11434/v1
|
||||
LocalLLMAPIKey string // optional; empty = no Authorization header sent
|
||||
LocalLLMModel string // the local server's model tag, e.g. huihui_ai/qwen3-abliterated:8b
|
||||
// LocalLLMTemp is the temperature the LOCAL adapter sends (Qwen3's non-thinking
|
||||
// 0.7), deliberately overriding the cascade's XAI_TEMPERATURE — ollama honours a
|
||||
// request temperature over its Modelfile, so passing the Grok-tuned 0.6 through
|
||||
// would silently mistune the local model (provider_local.go).
|
||||
LocalLLMTemp float64
|
||||
// LocalLLMMaxTok replaces MAX_OUTPUT_TOKENS on the local leg (0 = inherit).
|
||||
// ollama's max_tokens caps thinking+answer TOGETHER (unlike xAI, where thinking
|
||||
// bills on top of max_tokens), so a thinking route (reasoning_effort low/high) on
|
||||
// the cloud-sized 320 would burn the whole budget on reasoning and return empty
|
||||
// content. Local tokens are free — default roomier.
|
||||
LocalLLMMaxTok int
|
||||
// LocalLLMTimeout is the local leg's own request deadline: a half-alive tunnel
|
||||
// must fail fast enough to leave the shared RequestBudget room for the cloud
|
||||
// fallback (shorter than the transport's 60s per-attempt ceiling).
|
||||
LocalLLMTimeout time.Duration
|
||||
// LocalLLMHealthInterval is the background health-probe period (failover.go).
|
||||
LocalLLMHealthInterval time.Duration
|
||||
|
||||
SystemPromptPath string
|
||||
SystemPrompt string
|
||||
StateDir string
|
||||
|
|
@ -292,6 +319,8 @@ func LoadConfig() (*Config, error) {
|
|||
ReasoningEffort: strings.ToLower(strings.TrimSpace(getenv("REASONING_EFFORT", "high"))),
|
||||
GeminiBaseURL: strings.TrimRight(getenv("GEMINI_BASE_URL", "https://generativelanguage.googleapis.com/v1beta/openai"), "/"),
|
||||
GeminiModel: getenv("GEMINI_MODEL", "gemini-2.5-flash-lite"),
|
||||
LocalLLMBaseURL: strings.TrimRight(getenv("LOCAL_LLM_BASE_URL", ""), "/"),
|
||||
LocalLLMModel: getenv("LOCAL_LLM_MODEL", "huihui_ai/qwen3-abliterated:8b"),
|
||||
}
|
||||
|
||||
var problems []string
|
||||
|
|
@ -305,7 +334,8 @@ func LoadConfig() (*Config, error) {
|
|||
{"AS_TOKEN", &cfg.ASToken},
|
||||
{"HS_TOKEN", &cfg.HSToken},
|
||||
{"XAI_API_KEY", &cfg.XAIAPIKey},
|
||||
{"GEMINI_API_KEY", &cfg.GeminiAPIKey}, // optional; required only if a Gemini layer is on
|
||||
{"GEMINI_API_KEY", &cfg.GeminiAPIKey}, // optional; required only if a Gemini layer is on
|
||||
{"LOCAL_LLM_API_KEY", &cfg.LocalLLMAPIKey}, // optional; the local endpoint usually has no auth
|
||||
} {
|
||||
v, err := getSecret(s.key)
|
||||
if err != nil {
|
||||
|
|
@ -454,6 +484,52 @@ func LoadConfig() (*Config, error) {
|
|||
}
|
||||
cfg.Prices[cfg.ReasoningModel] = ModelPrice{InputPerM: rIn, CachedPerM: cfg.PriceCachedPerM, OutputPerM: rOut}
|
||||
|
||||
// Local LLM backend (voice failover, Phase 1).
|
||||
if cfg.LocalLLMEnabled, err = getenvBool("LOCAL_LLM_ENABLED", false); err != nil {
|
||||
problems = append(problems, err.Error())
|
||||
}
|
||||
if cfg.LocalLLMTemp, err = getenvFloat("LOCAL_LLM_TEMP", 0.7); err != nil {
|
||||
problems = append(problems, err.Error())
|
||||
}
|
||||
if cfg.LocalLLMMaxTok, err = getenvInt("LOCAL_LLM_MAX_OUTPUT_TOKENS", 1024); err != nil {
|
||||
problems = append(problems, err.Error())
|
||||
}
|
||||
var localTimeoutSec, localHealthSec int
|
||||
if localTimeoutSec, err = getenvInt("LOCAL_LLM_TIMEOUT_SECONDS", 45); err != nil {
|
||||
problems = append(problems, err.Error())
|
||||
}
|
||||
cfg.LocalLLMTimeout = time.Duration(localTimeoutSec) * time.Second
|
||||
if localHealthSec, err = getenvInt("LOCAL_LLM_HEALTH_INTERVAL_SECONDS", 15); err != nil {
|
||||
problems = append(problems, err.Error())
|
||||
}
|
||||
cfg.LocalLLMHealthInterval = time.Duration(localHealthSec) * time.Second
|
||||
// Local fail-fast (mirrors the Gemini block below): a half-configured local layer
|
||||
// must refuse to boot, not quietly serve everything from the paid cloud. The XAI
|
||||
// key stays REQUIRED regardless — the cloud is the fallback leg.
|
||||
if cfg.LocalLLMEnabled {
|
||||
if cfg.LocalLLMBaseURL == "" {
|
||||
problems = append(problems, "LOCAL_LLM_BASE_URL is required when LOCAL_LLM_ENABLED is set")
|
||||
}
|
||||
if cfg.LocalLLMModel == "" {
|
||||
problems = append(problems, "LOCAL_LLM_MODEL is required when LOCAL_LLM_ENABLED is set")
|
||||
}
|
||||
if localTimeoutSec <= 0 {
|
||||
problems = append(problems, "LOCAL_LLM_TIMEOUT_SECONDS must be > 0")
|
||||
}
|
||||
if localHealthSec <= 0 {
|
||||
problems = append(problems, "LOCAL_LLM_HEALTH_INTERVAL_SECONDS must be > 0")
|
||||
}
|
||||
// The $0 price entry is what books local answers as free: billing prices each
|
||||
// call by the model that ACTUALLY answered (LLMResponse.Model → computeUSD).
|
||||
// A name collision with a cloud model would zero THAT model's real price and
|
||||
// blind the daily ceiling — refuse it.
|
||||
if cfg.LocalLLMModel == cfg.XAIModel || cfg.LocalLLMModel == cfg.GeminiModel || cfg.LocalLLMModel == cfg.ReasoningModel {
|
||||
problems = append(problems, "LOCAL_LLM_MODEL must differ from XAI_MODEL / GEMINI_MODEL / REASONING_MODEL (its $0 price entry would zero the cloud model's billing)")
|
||||
} else if cfg.LocalLLMModel != "" {
|
||||
cfg.Prices[cfg.LocalLLMModel] = ModelPrice{} // free: home GPU, no per-token cost
|
||||
}
|
||||
}
|
||||
|
||||
// Fail-fast on broken cascade wiring (§5/F-FUNC-9), at EVERY start (not just
|
||||
// check-config): a layer that needs Gemini but has no key would silently never
|
||||
// fire. Better to refuse to start than to quietly run degraded.
|
||||
|
|
@ -605,6 +681,14 @@ func (c *Config) Summary() string {
|
|||
}
|
||||
return c.ProjectKBPath
|
||||
}()),
|
||||
fmt.Sprintf(" LOCAL_LLM = enabled=%t url=%s model=%s temp=%g max_out=%d timeout=%s health=%s key=%s",
|
||||
c.LocalLLMEnabled, func() string {
|
||||
if c.LocalLLMBaseURL == "" {
|
||||
return "(unset)"
|
||||
}
|
||||
return c.LocalLLMBaseURL
|
||||
}(), c.LocalLLMModel, c.LocalLLMTemp, c.LocalLLMMaxTok,
|
||||
c.LocalLLMTimeout, c.LocalLLMHealthInterval, redact(c.LocalLLMAPIKey)),
|
||||
" GEMINI_MODEL = " + c.GeminiModel,
|
||||
" GEMINI_API_KEY = " + redact(c.GeminiAPIKey),
|
||||
}, "\n")
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ package main
|
|||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// setBaseEnv sets the minimal valid environment (all cascade flags off) so each test
|
||||
|
|
@ -22,6 +23,9 @@ func setBaseEnv(t *testing.T) {
|
|||
"TRIVIAL_OFFLOAD_ENABLED", "WEB_ENABLED", "REASONING_ENABLED", "WEB_PROVIDER", "REASONING_MODEL",
|
||||
"WEB_PARANOID", "WEB_GROUNDING_DAILY_CAP", "GEMINI_GROUNDING_PER_PROMPT_USD",
|
||||
"PROJECT_KB_ENABLED", "PROJECT_KB_PATH",
|
||||
"LOCAL_LLM_ENABLED", "LOCAL_LLM_BASE_URL", "LOCAL_LLM_MODEL", "LOCAL_LLM_API_KEY",
|
||||
"LOCAL_LLM_API_KEY_FILE", "LOCAL_LLM_TEMP", "LOCAL_LLM_MAX_OUTPUT_TOKENS",
|
||||
"LOCAL_LLM_TIMEOUT_SECONDS", "LOCAL_LLM_HEALTH_INTERVAL_SECONDS",
|
||||
} {
|
||||
t.Setenv(k, "")
|
||||
}
|
||||
|
|
@ -41,7 +45,8 @@ func TestConfigAllCascadeFlagsDefaultOff(t *testing.T) {
|
|||
t.Fatalf("%v", err)
|
||||
}
|
||||
if cfg.RouterEnabled || cfg.RouterClassifierEnabled || cfg.TrivialOffloadEnabled ||
|
||||
cfg.WebEnabled || cfg.ReasoningEnabled || cfg.TelemetryEnabled || cfg.GrokPromptCache {
|
||||
cfg.WebEnabled || cfg.ReasoningEnabled || cfg.TelemetryEnabled || cfg.GrokPromptCache ||
|
||||
cfg.LocalLLMEnabled {
|
||||
t.Fatal("every cascade/telemetry flag must default off (cascade-off == today)")
|
||||
}
|
||||
if cfg.WebProvider != webProviderGrokWebSearch {
|
||||
|
|
@ -157,6 +162,60 @@ func TestConfigGeminiGroundingCapMustBePositive(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestConfigLocalLLMRequiresBaseURL: LOCAL_LLM_ENABLED without a base URL must refuse
|
||||
// to boot (the fail-fast contract), and a full local config registers the $0 price
|
||||
// entry for the local model with sane defaults for the rest.
|
||||
func TestConfigLocalLLMRequiresBaseURL(t *testing.T) {
|
||||
setBaseEnv(t)
|
||||
t.Setenv("LOCAL_LLM_ENABLED", "true")
|
||||
if _, err := LoadConfig(); err == nil || !strings.Contains(err.Error(), "LOCAL_LLM_BASE_URL") {
|
||||
t.Fatalf("want LOCAL_LLM_BASE_URL error, got %v", err)
|
||||
}
|
||||
|
||||
t.Setenv("LOCAL_LLM_BASE_URL", "http://172.18.0.1:11434/v1")
|
||||
cfg, err := LoadConfig()
|
||||
if err != nil {
|
||||
t.Fatalf("full local config should be valid: %v", err)
|
||||
}
|
||||
if cfg.LocalLLMModel != "huihui_ai/qwen3-abliterated:8b" {
|
||||
t.Fatalf("LOCAL_LLM_MODEL default = %q", cfg.LocalLLMModel)
|
||||
}
|
||||
if p, ok := cfg.Prices[cfg.LocalLLMModel]; !ok || p != (ModelPrice{}) {
|
||||
t.Fatalf("local model price = %+v (present=%t), want a $0 entry", p, ok)
|
||||
}
|
||||
if cfg.LocalLLMTemp != 0.7 || cfg.LocalLLMMaxTok != 1024 ||
|
||||
cfg.LocalLLMTimeout != 45*time.Second || cfg.LocalLLMHealthInterval != 15*time.Second {
|
||||
t.Fatalf("local defaults = temp %g / max %d / timeout %s / health %s, want 0.7/1024/45s/15s",
|
||||
cfg.LocalLLMTemp, cfg.LocalLLMMaxTok, cfg.LocalLLMTimeout, cfg.LocalLLMHealthInterval)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigLocalLLMModelCollision: naming the local model like a cloud one would
|
||||
// register a $0 price for the CLOUD model and blind the daily ceiling — refuse it.
|
||||
func TestConfigLocalLLMModelCollision(t *testing.T) {
|
||||
setBaseEnv(t)
|
||||
t.Setenv("LOCAL_LLM_ENABLED", "true")
|
||||
t.Setenv("LOCAL_LLM_BASE_URL", "http://172.18.0.1:11434/v1")
|
||||
t.Setenv("XAI_MODEL", "grok-4.20")
|
||||
t.Setenv("LOCAL_LLM_MODEL", "grok-4.20")
|
||||
if _, err := LoadConfig(); err == nil || !strings.Contains(err.Error(), "LOCAL_LLM_MODEL must differ") {
|
||||
t.Fatalf("want the model-collision error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigLocalLLMDisabledAddsNoPriceEntry: with the flag off nothing local leaks
|
||||
// into the price table (byte-identical config surface to today).
|
||||
func TestConfigLocalLLMDisabledAddsNoPriceEntry(t *testing.T) {
|
||||
setBaseEnv(t)
|
||||
cfg, err := LoadConfig()
|
||||
if err != nil {
|
||||
t.Fatalf("%v", err)
|
||||
}
|
||||
if _, ok := cfg.Prices[cfg.LocalLLMModel]; ok {
|
||||
t.Fatal("a disabled local backend must not register a price entry")
|
||||
}
|
||||
}
|
||||
|
||||
// The default per-prompt grounding fee is the paid-tier $0.035 (the operator must opt to 0).
|
||||
func TestConfigGroundingFeeDefault(t *testing.T) {
|
||||
setBaseEnv(t)
|
||||
|
|
|
|||
234
apps/ai-bot/failover.go
Normal file
234
apps/ai-bot/failover.go
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// failover.go is the local-first routing decorator over the LLMClient seam: prefer
|
||||
// the (free) home backend while it is reachable, fall back to the cloud transparently
|
||||
// when it is not. The cascade keeps calling b.llm.Complete and never learns which leg
|
||||
// answered — LLMResponse.Model carries that to billing/telemetry.
|
||||
//
|
||||
// Health is decided by an ACTIVE background probe plus a request-path circuit
|
||||
// breaker, both feeding one atomic healthy flag:
|
||||
//
|
||||
// - unhealthy → requests go straight to the cloud (no per-request timeout tax on a
|
||||
// dead tunnel);
|
||||
// - healthy → try local under its own short deadline (LOCAL_LLM_TIMEOUT_SECONDS —
|
||||
// a half-alive tunnel must fail fast enough to leave the shared RequestBudget
|
||||
// room for the cloud leg), and on a transport/5xx/timeout failure trip the
|
||||
// breaker and retry the SAME request on the cloud;
|
||||
// - recovery is probe-driven ONLY (anti-flapping): a tripped breaker needs
|
||||
// tripProbesToClose consecutive probe successes to close, so a tunnel that
|
||||
// accepts TCP but drops requests can't flap traffic between legs.
|
||||
//
|
||||
// A terminal 4xx from the local leg (except 429) does NOT fall back: that is a
|
||||
// config/request error (a wrong LOCAL_LLM_MODEL tag, a malformed body) and the cloud
|
||||
// would only mask it. 429 (a busy single-slot GPU) falls back like a timeout.
|
||||
// Per-request fallback logs at DEBUG; only state transitions and a both-legs failure
|
||||
// are louder.
|
||||
|
||||
const (
|
||||
// localProbeTimeout bounds one health probe. The probe is a bare GET /models —
|
||||
// answered from ollama's process memory, no inference — so anything slower than
|
||||
// this means the tunnel, not the model, is struggling.
|
||||
localProbeTimeout = 3 * time.Second
|
||||
// tripProbesToClose is how many consecutive probe successes re-close a breaker
|
||||
// tripped by a REQUEST failure. The probe can pass while real requests fail (a
|
||||
// cold model being loaded, a half-alive tunnel), so one success is not proof;
|
||||
// two spaced an interval apart also rate-limits the cold-load abort loop (an
|
||||
// aborted request cancels ollama's model load, keeping every next request cold).
|
||||
tripProbesToClose = 2
|
||||
// neverOnlineWarnAfter: consecutive probe failures — without the local leg EVER
|
||||
// having come online since boot — that trigger a one-time WARN. A wrong-valued
|
||||
// LOCAL_LLM_BASE_URL (say, missing the /v1 suffix) passes the boot presence
|
||||
// check but 404s forever, and per-request fallbacks log only at DEBUG — so
|
||||
// without this the enabled feature would be silently inoperative, serving 100%
|
||||
// of traffic from the paid cloud (the exact outcome the config fail-fast exists
|
||||
// to prevent).
|
||||
neverOnlineWarnAfter = 3
|
||||
)
|
||||
|
||||
type failoverLLMClient struct {
|
||||
primary LLMClient // local backend (free, preferred)
|
||||
fallback LLMClient // cloud (paid, always available)
|
||||
log *slog.Logger
|
||||
|
||||
legTimeout time.Duration // per-request deadline for the local leg
|
||||
probeURL string // <LOCAL_LLM_BASE_URL>/models
|
||||
probeKey string // optional bearer for the probe (usually empty)
|
||||
interval time.Duration // background probe period
|
||||
httpc *http.Client
|
||||
|
||||
// Breaker state. healthy starts false: until the first probe passes, traffic is
|
||||
// served by the cloud, so a boot with the home box offline never eats the probe
|
||||
// timeout on user requests.
|
||||
mu sync.Mutex
|
||||
healthy bool
|
||||
consecOK int // consecutive probe successes so far
|
||||
needOK int // successes required to close (1 normally, tripProbesToClose after a trip)
|
||||
consecFail int // consecutive probe failures (the never-online WARN counter)
|
||||
everHealthy bool // the leg has been online at least once since boot
|
||||
warnedDown bool // the one-time never-online WARN fired
|
||||
}
|
||||
|
||||
// NewFailoverLLMClient wraps primary (local) and fallback (cloud) into one LLMClient
|
||||
// and starts the background health probe, which lives until ctx is cancelled (the
|
||||
// process signal context — same lifetime as the bot).
|
||||
func NewFailoverLLMClient(ctx context.Context, primary, fallback LLMClient, cfg *Config, logger *slog.Logger) LLMClient {
|
||||
c := &failoverLLMClient{
|
||||
primary: primary,
|
||||
fallback: fallback,
|
||||
log: logger,
|
||||
legTimeout: cfg.LocalLLMTimeout,
|
||||
probeURL: cfg.LocalLLMBaseURL + "/models",
|
||||
probeKey: cfg.LocalLLMAPIKey,
|
||||
interval: cfg.LocalLLMHealthInterval,
|
||||
httpc: &http.Client{},
|
||||
needOK: 1,
|
||||
}
|
||||
// One INFO at construction, so the absence of the follow-up "online" line is a
|
||||
// documented signal (see the never-online WARN in noteProbe).
|
||||
logger.Info("local llm failover enabled; cloud serves until the first successful probe", "probe_url", c.probeURL)
|
||||
go c.probeLoop(ctx)
|
||||
return c
|
||||
}
|
||||
|
||||
func (c *failoverLLMClient) Complete(ctx context.Context, req LLMRequest) (*LLMResponse, error) {
|
||||
if !c.isHealthy() {
|
||||
c.log.DebugContext(ctx, "local llm unhealthy; serving from cloud")
|
||||
return c.fallback.Complete(ctx, req)
|
||||
}
|
||||
|
||||
lctx, cancel := context.WithTimeout(ctx, c.legTimeout)
|
||||
resp, err := c.primary.Complete(lctx, req)
|
||||
cancel()
|
||||
if err == nil {
|
||||
if strings.TrimSpace(resp.Text) != "" {
|
||||
return resp, nil
|
||||
}
|
||||
// A 2xx with EMPTY content from the local leg (thinking ate the whole token
|
||||
// budget, a content quirk of the abliterated model). On the plain cloud path
|
||||
// an empty completion is kept — it was billed — but the local one is free, so
|
||||
// retry the turn on the cloud instead of shipping the ⚠️-react empty path.
|
||||
// The backend is up and answering: not a breaker trip.
|
||||
c.log.DebugContext(ctx, "local llm returned empty content; retrying on cloud")
|
||||
return c.fallback.Complete(ctx, req)
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
// The WHOLE request budget is gone, not just the local leg — a cloud call
|
||||
// would die the same way, so don't burn a paid attempt; and don't trip the
|
||||
// breaker over our own cancellation.
|
||||
return nil, err
|
||||
}
|
||||
var se *httpStatusError
|
||||
if errors.As(err, &se) && se.status >= 400 && se.status < 500 && se.status != http.StatusTooManyRequests {
|
||||
return nil, err // terminal 4xx: a local config error — fail loud, never mask with the cloud
|
||||
}
|
||||
|
||||
c.trip()
|
||||
c.log.DebugContext(ctx, "local llm failed; falling back to cloud", "err", err)
|
||||
resp, ferr := c.fallback.Complete(ctx, req)
|
||||
if ferr != nil {
|
||||
// Both legs down — this is the only per-request failure loud enough for WARN
|
||||
// (the caller will still ERROR+react; this line preserves the local cause).
|
||||
c.log.WarnContext(ctx, "local llm and cloud fallback both failed", "local_err", err, "cloud_err", ferr)
|
||||
return nil, ferr
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func (c *failoverLLMClient) isHealthy() bool {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.healthy
|
||||
}
|
||||
|
||||
// trip opens the breaker after an in-request local failure. Recovery is probe-driven
|
||||
// only: requests never re-try the local leg until the prober has re-closed it.
|
||||
func (c *failoverLLMClient) trip() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.healthy {
|
||||
c.log.Info("local llm tripped offline by a request failure; serving from cloud until probes recover")
|
||||
}
|
||||
c.healthy = false
|
||||
c.consecOK = 0
|
||||
c.needOK = tripProbesToClose
|
||||
}
|
||||
|
||||
// noteProbe folds one probe result into the breaker. A failed probe opens it (the
|
||||
// tunnel is observably down — don't wait for a user request to find out) but does
|
||||
// not raise needOK: plain down/up recovers on the next success, only a request-path
|
||||
// trip demands the longer streak.
|
||||
func (c *failoverLLMClient) noteProbe(ok bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if !ok {
|
||||
if c.healthy {
|
||||
c.log.Info("local llm probe failed; serving from cloud")
|
||||
}
|
||||
c.healthy = false
|
||||
c.consecOK = 0
|
||||
c.consecFail++
|
||||
// Never came online since boot → the feature is silently inoperative (every
|
||||
// request quietly bills the cloud, visible only at DEBUG). Say so ONCE, loud.
|
||||
if !c.everHealthy && !c.warnedDown && c.consecFail >= neverOnlineWarnAfter {
|
||||
c.warnedDown = true
|
||||
c.log.Warn("local llm has never come online since boot; ALL traffic is served by the paid cloud — check LOCAL_LLM_BASE_URL and the tunnel", "probe_url", c.probeURL)
|
||||
}
|
||||
return
|
||||
}
|
||||
c.consecFail = 0
|
||||
c.consecOK++
|
||||
if !c.healthy && c.consecOK >= c.needOK {
|
||||
c.healthy = true
|
||||
c.everHealthy = true
|
||||
c.needOK = 1
|
||||
c.log.Info("local llm online; serving from local backend")
|
||||
}
|
||||
}
|
||||
|
||||
func (c *failoverLLMClient) probeLoop(ctx context.Context) {
|
||||
t := time.NewTicker(c.interval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
c.noteProbe(c.probe(ctx)) // first probe runs immediately, not an interval late
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// probe GETs the OpenAI models listing on the local base URL — cheap (no inference)
|
||||
// and it proves the whole path (SSH tunnel → relay → server) end to end, which is
|
||||
// exactly what dies when the home box goes away. It does NOT prove the model is
|
||||
// resident in VRAM; keeping it warm is OLLAMA_KEEP_ALIVE=-1's job (a cold model
|
||||
// answers slowly, which the request-path breaker handles).
|
||||
func (c *failoverLLMClient) probe(ctx context.Context) bool {
|
||||
pctx, cancel := context.WithTimeout(ctx, localProbeTimeout)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(pctx, http.MethodGet, c.probeURL, nil)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
if c.probeKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+c.probeKey)
|
||||
}
|
||||
resp, err := c.httpc.Do(req)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4<<10)) // drain for connection reuse
|
||||
return resp.StatusCode >= 200 && resp.StatusCode < 300
|
||||
}
|
||||
331
apps/ai-bot/failover_test.go
Normal file
331
apps/ai-bot/failover_test.go
Normal file
|
|
@ -0,0 +1,331 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// funcLLM is an LLMClient driven by a closure — for failover legs that need to
|
||||
// block, inspect their context, or fail in a specific typed way.
|
||||
type funcLLM struct {
|
||||
fn func(ctx context.Context, req LLMRequest) (*LLMResponse, error)
|
||||
calls int
|
||||
}
|
||||
|
||||
func (f *funcLLM) Complete(ctx context.Context, req LLMRequest) (*LLMResponse, error) {
|
||||
f.calls++
|
||||
return f.fn(ctx, req)
|
||||
}
|
||||
|
||||
// newTestFailover builds the decorator without the background prober, so tests
|
||||
// control the breaker state deterministically.
|
||||
func newTestFailover(primary, fallback LLMClient, healthy bool) *failoverLLMClient {
|
||||
return &failoverLLMClient{
|
||||
primary: primary,
|
||||
fallback: fallback,
|
||||
log: discardLog(),
|
||||
legTimeout: time.Second,
|
||||
healthy: healthy,
|
||||
needOK: 1,
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverHealthyPrefersLocal: with a healthy breaker the local leg answers and
|
||||
// the cloud is never touched; the response carries the local leg's Model untouched.
|
||||
func TestFailoverHealthyPrefersLocal(t *testing.T) {
|
||||
local := &fakeLLM{text: "local answer", model: "qwen-local"}
|
||||
cloud := &fakeLLM{text: "cloud answer", model: "grok-x"}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
|
||||
resp, err := c.Complete(context.Background(), LLMRequest{Model: "grok-x"})
|
||||
if err != nil {
|
||||
t.Fatalf("Complete: %v", err)
|
||||
}
|
||||
if resp.Text != "local answer" || resp.Model != "qwen-local" {
|
||||
t.Fatalf("resp = (%q, model %q), want the local leg's answer", resp.Text, resp.Model)
|
||||
}
|
||||
if local.calls != 1 || cloud.calls != 0 {
|
||||
t.Fatalf("calls local=%d cloud=%d, want 1/0", local.calls, cloud.calls)
|
||||
}
|
||||
if !c.isHealthy() {
|
||||
t.Fatal("a successful local answer must not touch the breaker")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverUnhealthyGoesStraightToCloud: an open breaker sends the request to the
|
||||
// cloud without paying the local leg's timeout (the local client is never called).
|
||||
func TestFailoverUnhealthyGoesStraightToCloud(t *testing.T) {
|
||||
local := &fakeLLM{text: "local"}
|
||||
cloud := &fakeLLM{text: "cloud answer", model: "grok-x"}
|
||||
c := newTestFailover(local, cloud, false)
|
||||
|
||||
resp, err := c.Complete(context.Background(), LLMRequest{})
|
||||
if err != nil {
|
||||
t.Fatalf("Complete: %v", err)
|
||||
}
|
||||
if resp.Text != "cloud answer" {
|
||||
t.Fatalf("resp = %q, want the cloud answer", resp.Text)
|
||||
}
|
||||
if local.calls != 0 || cloud.calls != 1 {
|
||||
t.Fatalf("calls local=%d cloud=%d, want 0/1", local.calls, cloud.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverTransportErrorFallsBackAndTrips: a network-class local failure answers
|
||||
// from the cloud within the SAME request and opens the breaker, so the next request
|
||||
// skips the local leg entirely.
|
||||
func TestFailoverTransportErrorFallsBackAndTrips(t *testing.T) {
|
||||
local := &fakeLLM{err: errors.New("dial tcp: connection refused")}
|
||||
cloud := &fakeLLM{text: "cloud answer", model: "grok-x"}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
|
||||
resp, err := c.Complete(context.Background(), LLMRequest{})
|
||||
if err != nil || resp.Text != "cloud answer" {
|
||||
t.Fatalf("resp=(%v,%v), want the cloud fallback", resp, err)
|
||||
}
|
||||
if c.isHealthy() {
|
||||
t.Fatal("a transport failure must trip the breaker")
|
||||
}
|
||||
if _, err := c.Complete(context.Background(), LLMRequest{}); err != nil {
|
||||
t.Fatalf("second Complete: %v", err)
|
||||
}
|
||||
if local.calls != 1 || cloud.calls != 2 {
|
||||
t.Fatalf("calls local=%d cloud=%d, want 1/2 (tripped breaker skips local)", local.calls, cloud.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailover5xxAnd429FallBack: retryable-class statuses (a dying tunnel's 502, a
|
||||
// busy single-slot GPU's 429) fall back to the cloud like a transport error.
|
||||
func TestFailover5xxAnd429FallBack(t *testing.T) {
|
||||
for _, status := range []int{500, 502, http.StatusTooManyRequests} {
|
||||
local := &fakeLLM{err: &httpStatusError{provider: "local", status: status, body: "boom"}}
|
||||
cloud := &fakeLLM{text: "cloud answer"}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
resp, err := c.Complete(context.Background(), LLMRequest{})
|
||||
if err != nil || resp.Text != "cloud answer" {
|
||||
t.Fatalf("status %d: resp=(%v,%v), want the cloud fallback", status, resp, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverTerminal4xxFailsLoud: a local 4xx is a config/request error (a wrong
|
||||
// LOCAL_LLM_MODEL tag) — masking it with a paid cloud answer would hide the
|
||||
// misconfiguration, so it surfaces as the request's error and the cloud is not called.
|
||||
func TestFailoverTerminal4xxFailsLoud(t *testing.T) {
|
||||
local := &fakeLLM{err: &httpStatusError{provider: "local", status: 404, body: "model not found"}}
|
||||
cloud := &fakeLLM{text: "cloud"}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
|
||||
if _, err := c.Complete(context.Background(), LLMRequest{}); err == nil {
|
||||
t.Fatal("want the local 4xx surfaced, got nil")
|
||||
}
|
||||
if cloud.calls != 0 {
|
||||
t.Fatalf("cloud called %d times on a local 4xx, want 0 (fail loud)", cloud.calls)
|
||||
}
|
||||
if !c.isHealthy() {
|
||||
t.Fatal("a 4xx is not an availability failure — the breaker must stay closed")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverBudgetGoneNoCloudRetry: when the WHOLE request budget is exhausted
|
||||
// (parent ctx done), the local error surfaces without burning a doomed cloud attempt
|
||||
// and without tripping the breaker over our own cancellation.
|
||||
func TestFailoverBudgetGoneNoCloudRetry(t *testing.T) {
|
||||
local := &funcLLM{fn: func(ctx context.Context, _ LLMRequest) (*LLMResponse, error) {
|
||||
<-ctx.Done()
|
||||
return nil, ctx.Err()
|
||||
}}
|
||||
cloud := &fakeLLM{text: "cloud"}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // the shared RequestBudget is gone before the call
|
||||
if _, err := c.Complete(ctx, LLMRequest{}); err == nil {
|
||||
t.Fatal("want an error when the request budget is gone")
|
||||
}
|
||||
if cloud.calls != 0 {
|
||||
t.Fatalf("cloud called %d times after budget exhaustion, want 0", cloud.calls)
|
||||
}
|
||||
if !c.isHealthy() {
|
||||
t.Fatal("our own cancellation must not trip the breaker")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverLocalTimeoutFallsBack: the local leg's own deadline (a half-alive
|
||||
// tunnel that accepts but never answers) expires and the request is served by the
|
||||
// cloud — the shared budget is still alive.
|
||||
func TestFailoverLocalTimeoutFallsBack(t *testing.T) {
|
||||
local := &funcLLM{fn: func(ctx context.Context, _ LLMRequest) (*LLMResponse, error) {
|
||||
<-ctx.Done() // blocks until the leg deadline fires
|
||||
return nil, ctx.Err()
|
||||
}}
|
||||
cloud := &fakeLLM{text: "cloud answer"}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
c.legTimeout = 10 * time.Millisecond
|
||||
|
||||
resp, err := c.Complete(context.Background(), LLMRequest{})
|
||||
if err != nil || resp.Text != "cloud answer" {
|
||||
t.Fatalf("resp=(%v,%v), want the cloud fallback after the leg timeout", resp, err)
|
||||
}
|
||||
if c.isHealthy() {
|
||||
t.Fatal("a local leg timeout must trip the breaker")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverEmptyLocalReplyRetriesOnCloud: a 2xx with empty content from the free
|
||||
// local leg (thinking ate the token budget) is retried on the cloud instead of
|
||||
// shipping the empty-completion ⚠️ path; the breaker stays closed (the backend is up).
|
||||
func TestFailoverEmptyLocalReplyRetriesOnCloud(t *testing.T) {
|
||||
local := &fakeLLM{text: " ", model: "qwen-local"}
|
||||
cloud := &fakeLLM{text: "cloud answer", model: "grok-x"}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
|
||||
resp, err := c.Complete(context.Background(), LLMRequest{})
|
||||
if err != nil || resp.Text != "cloud answer" {
|
||||
t.Fatalf("resp=(%v,%v), want the cloud retry on an empty local reply", resp, err)
|
||||
}
|
||||
if !c.isHealthy() {
|
||||
t.Fatal("an empty local reply is not an availability failure — no trip")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverBothLegsDown: local and cloud both failing surfaces the cloud error
|
||||
// (the caller refunds + reacts) — never a silent nil.
|
||||
func TestFailoverBothLegsDown(t *testing.T) {
|
||||
local := &fakeLLM{err: errors.New("tunnel down")}
|
||||
cloudErr := errors.New("xai down")
|
||||
cloud := &fakeLLM{err: cloudErr}
|
||||
c := newTestFailover(local, cloud, true)
|
||||
|
||||
_, err := c.Complete(context.Background(), LLMRequest{})
|
||||
if !errors.Is(err, cloudErr) {
|
||||
t.Fatalf("err = %v, want the cloud error surfaced", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverBreakerRecovery is the anti-flapping state machine: a request-path trip
|
||||
// needs tripProbesToClose consecutive probe successes to close; a plain probe-down
|
||||
// re-closes on the next success; a mid-streak probe failure resets the streak.
|
||||
func TestFailoverBreakerRecovery(t *testing.T) {
|
||||
c := newTestFailover(&fakeLLM{}, &fakeLLM{}, false)
|
||||
|
||||
// Startup: one successful probe brings the local leg online.
|
||||
c.noteProbe(true)
|
||||
if !c.isHealthy() {
|
||||
t.Fatal("one probe success must close the breaker at startup")
|
||||
}
|
||||
|
||||
// Request-path trip: one probe success is NOT enough, two are.
|
||||
c.trip()
|
||||
c.noteProbe(true)
|
||||
if c.isHealthy() {
|
||||
t.Fatal("after a trip one probe success must not re-close the breaker")
|
||||
}
|
||||
c.noteProbe(true)
|
||||
if !c.isHealthy() {
|
||||
t.Fatalf("after a trip %d probe successes must re-close the breaker", tripProbesToClose)
|
||||
}
|
||||
|
||||
// A mid-streak failure resets the streak.
|
||||
c.trip()
|
||||
c.noteProbe(true)
|
||||
c.noteProbe(false)
|
||||
c.noteProbe(true)
|
||||
if c.isHealthy() {
|
||||
t.Fatal("a probe failure mid-streak must reset the recovery counter")
|
||||
}
|
||||
c.noteProbe(true)
|
||||
if !c.isHealthy() {
|
||||
t.Fatal("a full consecutive streak after the reset must re-close the breaker")
|
||||
}
|
||||
|
||||
// Plain probe-down (no request trip): the next single success re-closes.
|
||||
c.noteProbe(false)
|
||||
if c.isHealthy() {
|
||||
t.Fatal("a probe failure must open the breaker")
|
||||
}
|
||||
c.noteProbe(true)
|
||||
if !c.isHealthy() {
|
||||
t.Fatal("plain probe-down must recover on one success")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverNeverOnlineWarnsOnce: a local leg that has never come online since
|
||||
// boot (a wrong LOCAL_LLM_BASE_URL 404s every probe) must WARN exactly once — an
|
||||
// enabled-but-inoperative feature silently billing the cloud is the failure the
|
||||
// fail-fast config comment promises to prevent — while a leg that WAS online logs
|
||||
// its down transition at INFO only.
|
||||
func TestFailoverNeverOnlineWarnsOnce(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
c := newTestFailover(&fakeLLM{}, &fakeLLM{}, false)
|
||||
c.log = slog.New(slog.NewTextHandler(&buf, nil))
|
||||
|
||||
for i := 0; i < neverOnlineWarnAfter-1; i++ {
|
||||
c.noteProbe(false)
|
||||
}
|
||||
if strings.Contains(buf.String(), "never come online") {
|
||||
t.Fatalf("WARN fired before %d consecutive failures: %s", neverOnlineWarnAfter, buf.String())
|
||||
}
|
||||
c.noteProbe(false)
|
||||
if n := strings.Count(buf.String(), "never come online"); n != 1 {
|
||||
t.Fatalf("never-online WARN count = %d, want exactly 1", n)
|
||||
}
|
||||
c.noteProbe(false) // still down — must not repeat
|
||||
if n := strings.Count(buf.String(), "never come online"); n != 1 {
|
||||
t.Fatalf("never-online WARN repeated: count = %d", n)
|
||||
}
|
||||
|
||||
// Once the leg HAS been online, a later long outage is a transition (INFO), not
|
||||
// the never-online WARN.
|
||||
buf.Reset()
|
||||
c.noteProbe(true)
|
||||
for i := 0; i < neverOnlineWarnAfter+1; i++ {
|
||||
c.noteProbe(false)
|
||||
}
|
||||
if strings.Contains(buf.String(), "never come online") {
|
||||
t.Fatalf("never-online WARN fired for a leg that was online: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TestFailoverProbeLoop exercises the real constructor end to end: the background
|
||||
// prober sees a live /models endpoint and brings the local leg online; the endpoint
|
||||
// dying takes it offline again.
|
||||
func TestFailoverProbeLoop(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/models" {
|
||||
t.Errorf("probe hit %q, want /models", r.URL.Path)
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
cfg := &Config{
|
||||
LocalLLMBaseURL: srv.URL,
|
||||
LocalLLMTimeout: time.Second,
|
||||
LocalLLMHealthInterval: 10 * time.Millisecond,
|
||||
}
|
||||
c := NewFailoverLLMClient(ctx, &fakeLLM{text: "local"}, &fakeLLM{text: "cloud"}, cfg, discardLog()).(*failoverLLMClient)
|
||||
|
||||
waitFor := func(want bool, what string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for c.isHealthy() != want {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("timed out waiting for %s", what)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
waitFor(true, "the prober to bring the local leg online")
|
||||
srv.Close()
|
||||
waitFor(false, "the prober to take the dead endpoint offline")
|
||||
}
|
||||
|
|
@ -219,7 +219,11 @@ func (c *openAIClient) attempt(ctx context.Context, payload []byte, reqHeaders m
|
|||
return nil, false, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+c.key)
|
||||
// A local backend (ollama / llama-server) usually runs without auth; an empty key
|
||||
// means "no Authorization header", not "Bearer " with an empty token.
|
||||
if c.key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+c.key)
|
||||
}
|
||||
for k, v := range c.headers {
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
|
|
@ -240,10 +244,10 @@ func (c *openAIClient) attempt(ctx context.Context, payload []byte, reqHeaders m
|
|||
logLLMExchange(ctx, c.log, c.name, payload, resp.StatusCode, data)
|
||||
|
||||
if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {
|
||||
return nil, true, fmt.Errorf("%s http %d: %s", c.name, resp.StatusCode, snippet(data))
|
||||
return nil, true, &httpStatusError{provider: c.name, status: resp.StatusCode, body: snippet(data)}
|
||||
}
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return nil, false, fmt.Errorf("%s http %d: %s", c.name, resp.StatusCode, snippet(data))
|
||||
return nil, false, &httpStatusError{provider: c.name, status: resp.StatusCode, body: snippet(data)}
|
||||
}
|
||||
|
||||
var out openAIResponse
|
||||
|
|
@ -258,6 +262,22 @@ func (c *openAIClient) attempt(ctx context.Context, payload []byte, reqHeaders m
|
|||
return &out, false, nil
|
||||
}
|
||||
|
||||
// httpStatusError is a non-2xx completion failure, carrying the status code as a
|
||||
// typed field so callers (the local/cloud failover) can classify it structurally —
|
||||
// a terminal 4xx is a config/request error that must fail loud, not be masked by a
|
||||
// fallback — instead of parsing the message. Error() keeps the exact wording the
|
||||
// plain fmt.Errorf produced, so logs and message-matching (isReasoningEffortUnsupported)
|
||||
// are unchanged.
|
||||
type httpStatusError struct {
|
||||
provider string
|
||||
status int
|
||||
body string
|
||||
}
|
||||
|
||||
func (e *httpStatusError) Error() string {
|
||||
return fmt.Sprintf("%s http %d: %s", e.provider, e.status, e.body)
|
||||
}
|
||||
|
||||
// isReasoningEffortUnsupported reports whether an xAI error is the specific 400 a
|
||||
// non-reasoning model returns when sent reasoning_effort ("...does not support parameter
|
||||
// reasoningEffort"). Matched loosely so both reasoning_effort and reasoningEffort spellings
|
||||
|
|
|
|||
|
|
@ -96,6 +96,30 @@ func TestCompleteReasoningEffortCachedAfterFirst(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestCompleteAuthHeaderOptional: an empty API key sends NO Authorization header (a
|
||||
// bare local endpoint — ollama/llama-server — has no auth), while a set key keeps the
|
||||
// exact Bearer header the cloud providers require.
|
||||
func TestCompleteAuthHeaderOptional(t *testing.T) {
|
||||
var gotAuth []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotAuth = append(gotAuth, r.Header.Get("Authorization"))
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
io.WriteString(w, `{"id":"ok","choices":[{"message":{"content":"hi"},"finish_reason":"stop"}],"usage":{}}`)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
req := openAIRequest{Model: "m", Messages: []openAIMessage{{Role: "user", Content: "hi"}}}
|
||||
if _, err := newOpenAIClient("local", srv.URL, "", nil, discardLog()).complete(context.Background(), req, nil); err != nil {
|
||||
t.Fatalf("keyless complete: %v", err)
|
||||
}
|
||||
if _, err := newOpenAIClient("xai", srv.URL, "sekret", nil, discardLog()).complete(context.Background(), req, nil); err != nil {
|
||||
t.Fatalf("keyed complete: %v", err)
|
||||
}
|
||||
if len(gotAuth) != 2 || gotAuth[0] != "" || gotAuth[1] != "Bearer sekret" {
|
||||
t.Fatalf("Authorization headers = %q, want [\"\" \"Bearer sekret\"]", gotAuth)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompleteTerminal4xxNoSelfHeal guards that the strip-and-retry is scoped to the
|
||||
// reasoning_effort 400 only: an unrelated 400 still fails fast (no spurious retry).
|
||||
func TestCompleteTerminal4xxNoSelfHeal(t *testing.T) {
|
||||
|
|
|
|||
|
|
@ -62,8 +62,15 @@ type LLMRequest struct {
|
|||
|
||||
// LLMResponse is a provider-neutral completion result.
|
||||
type LLMResponse struct {
|
||||
Text string
|
||||
Usage Usage
|
||||
Text string
|
||||
Usage Usage
|
||||
// Model is the model that ACTUALLY served this completion. It can differ from
|
||||
// LLMRequest.Model when a routing decorator (the local/cloud failover) answered
|
||||
// with a different backend than the cascade asked for — billing and telemetry
|
||||
// must follow the responder, not the request, or a free local answer books at
|
||||
// cloud prices. "" = unknown (an adapter predating this field, or a test fake);
|
||||
// consumers fall back to the requested model.
|
||||
Model string
|
||||
ProviderRequestID string // the backend's response id, logged for support/debug
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -86,6 +86,7 @@ func (c *geminiClient) Complete(ctx context.Context, req LLMRequest) (*LLMRespon
|
|||
CachedTokens: resp.Usage.PromptTokensDetails.CachedTokens,
|
||||
CompletionTokens: resp.Usage.CompletionTokens,
|
||||
},
|
||||
Model: req.Model, // Gemini serves exactly the requested model
|
||||
ProviderRequestID: resp.ID,
|
||||
}, nil
|
||||
}
|
||||
|
|
|
|||
90
apps/ai-bot/provider_local.go
Normal file
90
apps/ai-bot/provider_local.go
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
)
|
||||
|
||||
// provider_local.go is the thin adapter for the self-hosted inference backend (the
|
||||
// home ollama/llama-server behind the SSH tunnel — docs/plans/local_llm_backend.md).
|
||||
// Both speak the OpenAI Chat Completions wire format, so like xAI this is a shell
|
||||
// over the shared openAIClient transport (httpllm.go). It differs from the cloud
|
||||
// adapters in three deliberate ways:
|
||||
//
|
||||
// - The MODEL is the adapter's own (LOCAL_LLM_MODEL), never the cascade's: the
|
||||
// cascade asks for grok-* names that don't exist locally. The response's Model
|
||||
// reports what actually served, so billing prices it at the local $0 entry.
|
||||
// - The TEMPERATURE is the adapter's own (LOCAL_LLM_TEMP, Qwen3's non-thinking 0.7):
|
||||
// ollama honours a request temperature over the Modelfile, so passing through the
|
||||
// cascade's XAI_TEMPERATURE (0.6) would silently mistune the local model. The
|
||||
// remaining Qwen3 samplers (top_k/min_p/repeat_penalty) can't cross ollama's /v1
|
||||
// layer at all and are baked into the Modelfile at home.
|
||||
// - MAX TOKENS has a local override (LOCAL_LLM_MAX_OUTPUT_TOKENS): unlike xAI —
|
||||
// where thinking bills on top of max_tokens — ollama's cap covers thinking AND
|
||||
// answer together, so the cloud-sized MAX_OUTPUT_TOKENS would let a thinking
|
||||
// route (reasoning_effort low/high) burn the whole budget on reasoning and
|
||||
// return empty content. Local tokens are free; the override defaults roomier.
|
||||
//
|
||||
// ReasoningEffort passes through unchanged: ollama's /v1 maps it onto Qwen3 thinking
|
||||
// ("none" = off — the casual grok_direct route; "low"/"high" = on — verified live),
|
||||
// so the cascade's existing per-route efforts drive local thinking with no new knob.
|
||||
// ConvID (the x-grok-conv-id cache-routing header) and Tools are xAI concepts with no
|
||||
// local equivalent and are deliberately not forwarded.
|
||||
type localClient struct {
|
||||
http *openAIClient
|
||||
model string
|
||||
temp float64
|
||||
maxTok int // 0 = inherit the request's MaxTokens
|
||||
}
|
||||
|
||||
// NewLocalLLMClient builds the local-backend adapter. Returns the neutral LLMClient
|
||||
// so the bot holds no vendor type. The API key is optional (a bare local endpoint
|
||||
// sends no Authorization header — see openAIClient.attempt).
|
||||
func NewLocalLLMClient(cfg *Config, logger *slog.Logger) LLMClient {
|
||||
return &localClient{
|
||||
http: newOpenAIClient("local", cfg.LocalLLMBaseURL, cfg.LocalLLMAPIKey, nil, logger),
|
||||
model: cfg.LocalLLMModel,
|
||||
temp: cfg.LocalLLMTemp,
|
||||
maxTok: cfg.LocalLLMMaxTok,
|
||||
}
|
||||
}
|
||||
|
||||
func (c *localClient) Complete(ctx context.Context, req LLMRequest) (*LLMResponse, error) {
|
||||
msgs := make([]openAIMessage, len(req.Messages))
|
||||
for i, m := range req.Messages {
|
||||
msgs[i] = openAIMessage{Role: m.Role, Content: m.Content}
|
||||
}
|
||||
maxTok := req.MaxTokens
|
||||
if c.maxTok > 0 {
|
||||
maxTok = c.maxTok
|
||||
}
|
||||
var respFormat any
|
||||
if req.JSONOnly {
|
||||
respFormat = map[string]string{"type": "json_object"}
|
||||
}
|
||||
resp, err := c.http.complete(ctx, openAIRequest{
|
||||
Model: c.model,
|
||||
Messages: msgs,
|
||||
MaxTokens: maxTok,
|
||||
Temperature: c.temp,
|
||||
Stream: false,
|
||||
ReasoningEffort: req.ReasoningEffort,
|
||||
ResponseFormat: respFormat,
|
||||
}, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &LLMResponse{
|
||||
Text: resp.Text(),
|
||||
Usage: Usage{
|
||||
PromptTokens: resp.Usage.PromptTokens,
|
||||
CachedTokens: resp.Usage.PromptTokensDetails.CachedTokens,
|
||||
CompletionTokens: resp.Usage.CompletionTokens,
|
||||
// ReasoningTokens deliberately left 0: ollama counts thinking INSIDE
|
||||
// completion_tokens (subset semantics — see llm.go), and the details
|
||||
// field is absent anyway. It's all $0 regardless.
|
||||
},
|
||||
Model: c.model, // the local model answered, whatever the cascade asked for
|
||||
ProviderRequestID: resp.ID,
|
||||
}, nil
|
||||
}
|
||||
103
apps/ai-bot/provider_local_test.go
Normal file
103
apps/ai-bot/provider_local_test.go
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const localOKBody = `{"id":"local-1","model":"qwen-test",` +
|
||||
`"choices":[{"message":{"content":"привет"},"finish_reason":"stop"}],` +
|
||||
`"usage":{"prompt_tokens":10,"completion_tokens":5}}`
|
||||
|
||||
func localTestCfg(base string) *Config {
|
||||
return &Config{
|
||||
LocalLLMBaseURL: base,
|
||||
LocalLLMModel: "qwen-test",
|
||||
LocalLLMTemp: 0.7,
|
||||
LocalLLMMaxTok: 1024,
|
||||
}
|
||||
}
|
||||
|
||||
// TestLocalClientOverridesModelTempMaxTokens: the adapter's contract with the local
|
||||
// backend — its OWN model tag and Qwen3 temperature (never the cascade's Grok name /
|
||||
// XAI_TEMPERATURE), the roomier local max_tokens, reasoning_effort passed through,
|
||||
// and no xAI-specific conv-id header. The response reports the local model as the
|
||||
// responder (the billing/telemetry key).
|
||||
func TestLocalClientOverridesModelTempMaxTokens(t *testing.T) {
|
||||
var gotBody map[string]any
|
||||
var gotConvHeader string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
if err := json.Unmarshal(body, &gotBody); err != nil {
|
||||
t.Errorf("request body: %v", err)
|
||||
}
|
||||
gotConvHeader = r.Header.Get("x-grok-conv-id")
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
io.WriteString(w, localOKBody)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
c := NewLocalLLMClient(localTestCfg(srv.URL), discardLog())
|
||||
resp, err := c.Complete(context.Background(), LLMRequest{
|
||||
Model: "grok-4.20", // what the cascade asks for — must NOT reach the wire
|
||||
Messages: []Message{{Role: "user", Content: "привет"}},
|
||||
MaxTokens: 320,
|
||||
Temperature: 0.6, // the Grok tuning — must NOT reach the wire
|
||||
ReasoningEffort: "none",
|
||||
ConvID: "conv-1", // xAI cache routing — no local equivalent
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Complete: %v", err)
|
||||
}
|
||||
if gotBody["model"] != "qwen-test" {
|
||||
t.Fatalf("wire model = %v, want the adapter's own qwen-test", gotBody["model"])
|
||||
}
|
||||
if gotBody["temperature"] != 0.7 {
|
||||
t.Fatalf("wire temperature = %v, want the local 0.7 (not the cascade's 0.6)", gotBody["temperature"])
|
||||
}
|
||||
if gotBody["max_tokens"] != float64(1024) {
|
||||
t.Fatalf("wire max_tokens = %v, want the local override 1024", gotBody["max_tokens"])
|
||||
}
|
||||
if gotBody["reasoning_effort"] != "none" {
|
||||
t.Fatalf("wire reasoning_effort = %v, want the cascade's value passed through", gotBody["reasoning_effort"])
|
||||
}
|
||||
if gotConvHeader != "" {
|
||||
t.Fatalf("x-grok-conv-id = %q sent to the local backend, want none", gotConvHeader)
|
||||
}
|
||||
if resp.Model != "qwen-test" {
|
||||
t.Fatalf("resp.Model = %q, want qwen-test (the responder identity)", resp.Model)
|
||||
}
|
||||
if resp.Text != "привет" || resp.Usage.PromptTokens != 10 || resp.Usage.CompletionTokens != 5 {
|
||||
t.Fatalf("resp = %+v, want the parsed local completion", resp)
|
||||
}
|
||||
if resp.Usage.ReasoningTokens != 0 {
|
||||
t.Fatal("local ReasoningTokens must stay 0 (subset semantics — see llm.go)")
|
||||
}
|
||||
}
|
||||
|
||||
// TestLocalClientInheritsMaxTokensWhenUnset: LOCAL_LLM_MAX_OUTPUT_TOKENS=0 means
|
||||
// "inherit the request's MaxTokens".
|
||||
func TestLocalClientInheritsMaxTokensWhenUnset(t *testing.T) {
|
||||
var gotBody map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
_ = json.Unmarshal(body, &gotBody)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
io.WriteString(w, localOKBody)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
cfg := localTestCfg(srv.URL)
|
||||
cfg.LocalLLMMaxTok = 0
|
||||
c := NewLocalLLMClient(cfg, discardLog())
|
||||
if _, err := c.Complete(context.Background(), LLMRequest{MaxTokens: 320}); err != nil {
|
||||
t.Fatalf("Complete: %v", err)
|
||||
}
|
||||
if gotBody["max_tokens"] != float64(320) {
|
||||
t.Fatalf("wire max_tokens = %v, want the inherited 320", gotBody["max_tokens"])
|
||||
}
|
||||
}
|
||||
|
|
@ -59,6 +59,7 @@ func (c *xaiClient) Complete(ctx context.Context, req LLMRequest) (*LLMResponse,
|
|||
// xAI bills reasoning at the output rate on top of completion_tokens.
|
||||
ReasoningTokens: resp.Usage.CompletionTokensDetails.ReasoningTokens,
|
||||
},
|
||||
Model: req.Model, // xAI serves exactly the requested model
|
||||
ProviderRequestID: resp.ID,
|
||||
}, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -161,7 +161,7 @@ func (b *Bot) routeLayer1(ctx context.Context, rcx string, l0 rd.Layer0, cost *C
|
|||
if err != nil {
|
||||
return RouterDecision{}, err
|
||||
}
|
||||
cost.Router += computeUSD(b.cfg.GeminiModel, resp.Usage, b.cfg)
|
||||
cost.Router += computeUSD(servedModel(b.cfg.GeminiModel, resp), resp.Usage, b.cfg)
|
||||
|
||||
// The classifier schema IS routedecide.Verdict (tagged), so unmarshal straight into it.
|
||||
var v rd.Verdict
|
||||
|
|
|
|||
|
|
@ -66,8 +66,34 @@ then dispatches; **any layer off or failing degrades to `grok_direct`** (never a
|
|||
[llm.go](../../apps/ai-bot/llm.go) (`Message`/`Usage`/`LLMRequest`/`LLMResponse`/`LLMClient`) +
|
||||
[httpllm.go](../../apps/ai-bot/httpllm.go) (shared OpenAI-compatible transport + retry) + thin
|
||||
adapters [provider_xai.go](../../apps/ai-bot/provider_xai.go) /
|
||||
[provider_gemini.go](../../apps/ai-bot/provider_gemini.go) + [pricing.go](../../apps/ai-bot/pricing.go)
|
||||
[provider_gemini.go](../../apps/ai-bot/provider_gemini.go) /
|
||||
[provider_local.go](../../apps/ai-bot/provider_local.go) + [pricing.go](../../apps/ai-bot/pricing.go)
|
||||
(`priceFor` model→price map). `Bot.llm` is an `LLMClient`, never a concrete vendor type.
|
||||
**The seam is the extension point** — one adapter per provider per file, cross-provider policy
|
||||
(failover, billing, telemetry) lives in decorators/neutral fields, never in an adapter:
|
||||
`LLMResponse.Model` reports which model **actually** answered, and every `computeUSD` call-site
|
||||
bills by it (`servedModel`), so routing decorators can swap backends without touching accounting.
|
||||
|
||||
## Local LLM backend (voice failover — Phase 1 of `docs/plans/local_llm_backend.md`)
|
||||
|
||||
`LOCAL_LLM_ENABLED` wraps `b.llm` in [failover.go](../../apps/ai-bot/failover.go): primary = the
|
||||
self-hosted backend ([provider_local.go](../../apps/ai-bot/provider_local.go), any OpenAI-compatible
|
||||
server — ollama / llama.cpp / vLLM / LM Studio — behind `LOCAL_LLM_BASE_URL`; in prod the home
|
||||
GPU over an SSH reverse-tunnel relay at `http://172.18.0.1:11434/v1`), fallback = cloud Grok.
|
||||
**Every voice call** (grok_direct, reason, web/project synthesis) goes local-first while an active
|
||||
`GET /models` probe + circuit breaker say the tunnel is healthy; otherwise — or on a local
|
||||
transport/5xx/timeout/429 failure mid-request — the same request is served by Grok. Recovery is
|
||||
probe-driven only (a request-path trip needs 2 consecutive probe passes — anti-flapping); a local
|
||||
**terminal 4xx fails loud** (config error, never masked by paid cloud answers); an **empty** local
|
||||
2xx retries on the cloud (free leg, so no billed-empty dilemma). A local answer books **$0**
|
||||
(`cfg.Prices[LOCAL_LLM_MODEL]={0,0,0}` + billing by `LLMResponse.Model`); `request_log.models.final`
|
||||
records the responder, so the local share and quality (reaction feedback) are measurable per
|
||||
backend. The adapter sends its OWN model/temperature (`LOCAL_LLM_TEMP=0.7`, Qwen3 non-thinking)
|
||||
and a roomier `LOCAL_LLM_MAX_OUTPUT_TOKENS=1024` (ollama's cap covers thinking+answer TOGETHER,
|
||||
unlike xAI where thinking bills on top — the cloud 320 would return empty content on thinking
|
||||
routes); `reasoning_effort` passes through (ollama `/v1`: `none`=off, `low`/`high`=thinking on).
|
||||
Classifier (`b.gemini`) and web fetch (`b.web`) stay cloud — phases 2–3. Env table in the
|
||||
[README](../../apps/ai-bot/README.md#local-llm-backend-voice-failover-default-off).
|
||||
|
||||
## Money, invariants & store ([store.go](../../apps/ai-bot/store.go))
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue