textmachine/backend/internal/runevents/runevents.go

427 lines
26 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

// Package runevents is the ENGINE side of the run-event seam (row 103): the vocabulary of the
// NDJSON stream the platform tails, and the append-only journal it is written to.
//
// The transport is ratified and not open (D39.106 §2, research/25 §Форма): the engine is a transient
// systemd unit per run, the platform is NOT its parent, and the stream is `events.jsonl` in the BOOK's
// directory — an outbox projection of rows the engine has already committed to its SQLite. Delivery is
// at-least-once; a re-read line is normal (D39.119 п.3 / PD-105). What this package owns is only the
// FORM: the envelope, the payloads and the file discipline. WHEN an event happens is the driver's
// (internal/pipeline), and the durable sequencing is the store's (internal/store, events_outbox).
//
// The payload shapes mirror the platform's reader (`platform/internal/ingest/events.go`), which is the
// platform's PROPOSAL written as code, "so the engine zone can answer it with a diff". Where this file
// differs from that one, the difference is DELIBERATE and named in the doc comment of the type:
// `Ceiling` adds `scope` and `Finished` widens its outcome vocabulary. (`Progress` was going to omit
// `eta_seconds`; measurement showed omission NULLs the consumer's column, so it is carried — see there.)
//
// Nothing here is language-, pair- or book-specific, and nothing may become so: an event carries
// counters, ordinals and engine-side enums only (общность §0.1 — a pair that is not in the repository
// must stream identically without a line of Go changing).
package runevents
import (
"encoding/json"
"fmt"
"math"
"time"
)
// StreamVersion is the version of the stream this build writes. The rule is terraform's, ratified by
// D39.85: a MINOR bump adds fields and event types — a reader ignores the ones it does not know; a
// MAJOR bump is refused by the reader outright. So: adding a field or an event type bumps the minor,
// changing what an existing field MEANS bumps the major.
//
// 1.1 added a field (`Ceiling.Scope`) and two outcome values, and the bump was the point rather than a
// formality: a version is a fact about the BYTES on the wire, not about whether the vocabulary diff has
// been accepted yet. Leaving it behind would make two streams that differ in content claim the same
// version, which is the one thing a version exists to prevent; the reader compares only the major, so a
// minor bump costs nothing and is safe in both directions.
//
// 1.2 adds `Finished.Volume` — the run's delivery ledger, seven counters saying what a volume grant
// actually bought (D39.181 п.2). By the rule above that is a minor, and it is the same reasoning applied
// twice: the field is present on the wire whether or not a reader knows it, so the version has to say so.
//
// 1.3 adds `Ceiling.ShortfallMicroUSD` and `Finished.Money` — HOW MUCH WAS MISSING, and what the run
// bought before it ran out (backlog rows 277/278). Until now the run printed the missing amount to the
// stderr of a transient unit and it died there, so a buyer whose book stopped could be told that it
// stopped and nothing about what would restart it. A minor by the same rule, and by the same reasoning:
// the bytes on the wire changed, so the version has to say so — the miss recorded below is exactly this
// omission, made once already.
//
// ⚠ IT WAS ALMOST NOT BUMPED, and the miss is worth keeping visible: the money-and-honesty pack added the
// field and left the constant at 1.1 — publishing a fact whose own stream version denied it existed,
// inside a pack about surfaces that do not say what they know. Caught by acceptance. The platform's
// mirror constant is NOT touched from here: the major is unchanged, so its reader is unaffected, and
// editing another zone's copy is what backlog row 246 is about.
const StreamVersion = "1.3"
// Type is the event name.
type Type string
const (
// TypeHello is always the first line a process writes: the version handshake.
TypeHello Type = "hello"
// TypeProgress carries the per-phase counters. Per phase because a unit is done only once its edit
// resolved, so one end-to-end counter reads zero for the whole draft wave (row 99 / D39.122 п.2б).
TypeProgress Type = "progress"
// TypeUnitDone is one resolved output unit.
TypeUnitDone Type = "unit_done"
// TypeBankStop is the book-wide signing stop before the edit wave.
TypeBankStop Type = "bank_stop"
// TypeCeiling is the resumable halt on a spend ceiling: the fact, and how much was missing.
TypeCeiling Type = "ceiling"
// TypeSpend is the cumulative spend counter.
TypeSpend Type = "spend"
// TypeFinished is the terminal line of a run that ended on purpose.
TypeFinished Type = "finished"
)
// Envelope is one line of the stream. Seq is per PROCESS and starts at 1: a resumed run is a NEW
// process that appends a second hello to the same file and numbers from 1 again, which is why the
// ratified idempotency key is (engine_run_id, seq) and not the platform's run id.
type Envelope struct {
Seq int64 `json:"seq"`
Type Type `json:"type"`
Time time.Time `json:"time"`
Data json.RawMessage `json:"data"`
}
// Hello is the handshake payload.
type Hello struct {
StreamVersion string `json:"stream_version"`
// EngineRunID is this PROCESS's identity — the trace id, which since row 102 the caller may supply
// (TM_TRACE_ID). It must be non-empty: it is half of the idempotency key, and an empty one would
// collapse every run's events into one namespace instead of failing.
EngineRunID string `json:"engine_run_id"`
BookID string `json:"book_id"`
// ChunkerVersion lets a reader notice that the chapter manifest it persisted was produced by a
// different chunker — the case that silently re-numbers chapters.
ChunkerVersion string `json:"chunker_version"`
}
// Counter is one phase's done/total pair, in OUTPUT UNITS — the granularity every engine read model
// counts in (status.go, the manifest). Counting the draft wave in CHUNKS instead would put the stream
// and the `status --json` resync on two different scales, and the platform folds both into one column.
type Counter struct {
Done int `json:"done"`
Total int `json:"total"`
}
// Progress is the run's per-wave counters. Total is 0 for a wave this pipeline does not have, which is
// how a reader tells "no such phase" from "none of it is done yet".
//
// ETASeconds is carried, and the reason is worth recording because the first version of this pack
// omitted it on an argument that MEASUREMENT destroyed. The argument was: the engine has one ETA
// definition already (pipeline/status.go, a book-lifetime mean over stored latencies), computing it here
// would cost a store aggregate per unit, and the field is optional to the reader anyway. The last clause
// is false. The consumer's progress handler ASSIGNS the column unconditionally —
// `update runs set … eta_seconds = $6` with `etaOrNil(p.ETASeconds)` (platform/internal/pgstore/sink.go)
// — so an absent field decodes to 0 and NULLS the estimate on every single progress line, erasing what
// the `status --json` resync had just written. Omitting a field is not leaving it alone.
//
// So it is emitted, computed from THIS RUN's own throughput (see pipeline/events.go). That deliberately
// differs from status.go's book-lifetime mean: this one is what the run is achieving now, it costs no
// query, and the alternative on the table was not "a second opinion" but "no estimate at all".
type Progress struct {
Draft Counter `json:"draft"`
Edit Counter `json:"edit"`
ETASeconds int `json:"eta_seconds,omitempty"`
}
// UnitDone is one resolved output unit. Chapter is the engine's dense 1-based ordinal and Unit is the
// unit's LEADER chunk index — together the join key the manifest publishes as `first_chunk_idx`, which
// is how a reader maps this onto its own opaque ids.
//
// Shipped and Flagged are BOTH carried and neither implies the other: a flagged unit legally ships text
// (a cosmetic sanitizer strip, a c-lite member drop), and the pair is exactly the contract's derivation
// of unit state.
//
// It is emitted ONLY for a unit THIS process resolved — a unit already resolved when the process started
// is walked again at $0 on every resume and re-announcing it would make a counting reader count it twice
// (see pipeline/events.go, `resolvedAtStart`).
type UnitDone struct {
Chapter int `json:"chapter"`
Unit int `json:"unit"`
Wave string `json:"wave"` // draft | edit
Shipped bool `json:"shipped"`
Flagged bool `json:"flagged"`
// Reason is the ENGINE's flag reason (glossary_miss, sanitizer_stripped, …) — stored by a reader,
// never projected verbatim, so a reason it has never heard of still gets a neutral phrase.
Reason string `json:"reason,omitempty"`
}
// BankStop is the signing stop. The full table travels as an artifact (row 101), not through the
// stream: a thousand rows are not an event.
type BankStop struct {
TermsProposed int `json:"terms_proposed"`
}
// Ceiling is the ceiling halt. It carries the FACT and nothing else: money never reaches the platform's
// wire or its INFO logs (D39.84), and the stop is resumable, so it is NOT a failure — which is the whole
// of PD-113.
//
// ⚠ DIFF against the platform's proposal: `scope` is added. It is not money — it names WHICH ceiling
// stopped the run, book or day — and it closes the diagnosis half of PD-157: a book whose `day_usd` the
// platform never chose stops the run on a limit the platform cannot even see, and today it cannot tell
// that from the ceiling it set itself.
type Ceiling struct {
Halted bool `json:"halted"`
Scope string `json:"scope"` // book | day
// ShortfallMicroUSD is HOW MUCH WAS MISSING: the amount by which the engine's admission arithmetic
// overshot the ceiling — top up by at least this and the same call is admitted. Integer micro-USD,
// rounded UP (money never travels as a float, PD-79, and a figure a person tops up AGAINST must never
// be short).
//
// ⛔ RATIFIED BY THE OWNER — D39.203, 05.09: the engine MAY tell the platform how much to add. What
// the decision rests on is a distinction, and the distinction is what makes this field legitimate
// rather than tolerated: **what leaves is a PRE-CALL ESTIMATE — the reservation the engine asks for
// BEFORE it dials — and not a COST, what was actually charged after.** ПТ-35 bans the cost and stands
// UNNARROWED; a settlement still does not leave this process by any door.
//
// ⚠ HOW IT GOT DECIDED IS WORTH KEEPING, because two sessions in a row closed it wrongly on their own
// authority. An earlier version of this comment called it «an exception taken knowingly» and cited a
// ratification that does not exist in the decisions log; the version after that marked it
// `pending owner`, which was right. The ban is the OWNER's (ПТ-35, `docs/product-requirements.md`):
// revoking «no money on screen» on 05.09 left the rest standing in so many words — «цены моделей,
// стоимость стадий И ВЫЗОВОВ» — and the pack that ordered this field applied that ban to the WIRE,
// not to a screen (`docs/BACKEND_MONEYSTOP_SESSION_PROMPT.md:78`). So the measurement below never
// disproved this field's implementation; it disproved a PREMISE OF THE ORDER, and a premise is the
// owner's to re-decide, not a session's.
//
// ⚠ AND THE FORM WILL CHANGE: the owner chose «how much to add so ANY next call passes»
// (`max(shortfall, step_max headroom)`), which makes `shortfall_micro_usd` a false name and carries
// a `StreamVersion` minor with it. That is a NEW ORDER, not a defect here — this field is not wrong,
// it is narrower: it says how much was missing for THIS call. D39.203 §56.
//
// ⛔ IT DOES DISCLOSE THE ESTIMATE OF THE REFUSED CALL — the thing D39.203 permits. An earlier
// version of this comment claimed the opposite — «discloses nothing about what any one call cost» —
// and acceptance disproved it with a live probe rather than an argument: the admission arithmetic is
// `committed + reserved + estimate > ceiling`, so a reader who knows the ceiling (the platform set it)
// and the cumulative committed spend (this stream publishes it — see Spend) recovers
// `estimate = shortfall + ceiling committed reserved`. On the ORDINARY terminal refusal the sky is
// empty by construction (the run stops through waitNothingInFlight), so `reserved` is nil-to-leftover
// and the recovered figure lands within a rounding micro-dollar of the engine's own
// `denied estimate=$…`. Measured, not reasoned.
//
// ⚠ AND THE ARGUMENT THAT WAS OFFERED FOR IT WAS FALSE TOO, so it is retired here rather than
// repeated: «the same door is already open through `step_max_usd`». No CONSUMER has it open —
// `git grep -E "StepMax|step_max" -- platform/` is EMPTY and the identifier appears nowhere in this
// package. But the ENGINE does publish it, because the same pack ordered it into `manifest --json`,
// and `step_max_usd` is by construction the estimate of ONE call — the most expensive single
// reservation the book can ask for. So the order itself holds both halves: item (3) forbids a call's
// price leaving, item (4) requires the largest one to be published. That tension is the owner's to
// resolve, and naming it is the honest form of «the same door».
//
// WHAT IS NOT DISCLOSED EITHER WAY, so that the open question stays the narrow one: the STRUCTURE of
// the costs — what a model charges, what a stage costs, how a call divides between prompt and
// completion — and what a call actually COST, since both this figure and `step_max_usd` are pre-call
// ESTIMATES, never a settlement. And what the field buys, which is why it was ordered: a stop that
// cannot say how much was missing leaves a buyer with a dead book and no next step — measured live on
// 04.09.
//
// ⚠ ONE PATH KEEPS THE PRICE: a stop taken through settleCannotHelp can leave calls IN FLIGHT, and
// then `reserved` is neither zero nor known to the consumer, so the subtraction yields a bound rather
// than the estimate.
//
// OMITTED when the engine cannot state it honestly: a `day` scope stop (that ceiling sums every book
// in the store while the engine's figures are one book's), or a ledger read that failed at the moment
// of the refusal. Absent means «not stated», never «nothing was missing» — a stated shortfall is
// always at least one micro-USD.
ShortfallMicroUSD int64 `json:"shortfall_micro_usd,omitempty"`
}
// Spend is the freshness channel for money and ONLY that: the balance is protected by the platform's
// hold and by the per-book ceiling the engine enforces itself, so a lost tail costs an indicator its
// accuracy and never costs an account its correctness. Building enforcement on this event is forbidden —
// the stream is at-least-once and a crash truncates it (D39.106 §2: холд+потолок = защита, события =
// свежесть).
//
// CUMULATIVE, not a delta, so a re-delivered line is harmless to a reader that keeps the maximum. It is
// the book's LIFETIME committed spend — the same figure `status --json` reports as `committed_usd`, so
// the two channels can never quote different numbers. Integer micro-USD: money never travels as a float
// (PD-79), and the engine's ledger is a lower bound, so the conversion rounds UP.
type Spend struct {
CommittedMicroUSD int64 `json:"committed_micro_usd"`
}
// Finished is the terminal line of a run that ended on purpose. Its absence is meaningful: a stream
// without it ended without finishing a book run — a crash, or a command that never started one.
//
// ⚠ DIFF against the platform's proposal: the outcome vocabulary gains `ceiling` and `stopped`. The
// proposal has clean|flagged|bank_stop|failed, and a ceiling halt fits none of them — `failed` is exactly
// what the contract forbids for a resumable stop (PD-113), and leaving the stream unterminated would make
// "stopped on purpose" indistinguishable from "truncated by a crash", which is the same confusion one
// level down. `stopped` is its sibling for a caught SIGTERM (PD-152).
type Finished struct {
Outcome string `json:"outcome"`
// Volume is the run's DELIVERY LEDGER, present when a volume grant was in force AND the run has
// something to say about it: work was held back, or work was done OUTSIDE the grant (the engine
// finishes units an earlier run began without charging its own grant for them — backend volume.go).
//
// ⛔ WHY ITS PRESENCE IS THE SIGNAL, and why there is no `volume` OUTCOME. A run stopped by the volume
// ceiling did what it was bought to do, so it ends `clean` (or `flagged`, if a unit needed a human) and
// exits 0 or 2 — machine-identical to a run that reached the end of the book. The engine knows the
// difference and, until this field, said it in ONE PROSE LINE that no consumer of this stream reads.
//
// A new OUTCOME value was the obvious fix and is the wrong one. The outcome vocabulary is deliberately
// readable from either channel — «one fact travelling twice» — and every value it has maps to an exit
// code. A value living only in the stream would make the two channels report DIFFERENT outcomes for one
// run: the line would say `volume` while OutcomeOf(0) said `clean`. That is not an extension of the
// vocabulary, it is a contradiction inside it. A new exit code is not available either — the top-level
// band is frozen. Numbers have no vocabulary to contradict, so the fact travels as numbers.
//
// ⚠ WHAT ITS PRESENCE MEANS, exactly — the original rule was «the grant held something back», and that
// is now only one of the two reasons. A run granted more than the book had left, which also did nothing
// beyond its grant, is an ordinary completion and carries nothing here; a run whose grant held units
// back carries this; and so does a run that reached the end of the book while finishing units an
// earlier run began, because it delivered MORE than it was granted and no other field on this frame
// explains that. So presence means «a grant was in force and this run has something to report about
// it», and `left_fresh`/`left_rework` are what tell the two apart: both zero means the book ended.
// ⚠ The count of units finished outside the grant is NOT on this frame — the engine knows it, and
// adding a field here is a contract change the owner declined on 03.09 («оставить как есть»). Until
// that is revisited, `delivered` can exceed `max_units` and the split behind it is not on the wire.
//
// Ratified with the disclosure law, D39.181 п.2; the presence rule widened when the engine learned not
// to charge a grant twice for one unit (backlog row 232).
Volume *VolumeLedger `json:"volume,omitempty"`
// Money is what a run whose SPEND CEILING WAS REACHED actually bought.
//
// ⛔ ITS PRESENCE RULE IS «A CEILING WAS REACHED», NOT «THE OUTCOME IS `ceiling`», AND THE DIFFERENCE
// IS THE WHOLE OF THIS PARAGRAPH. An earlier version of it said «present only on `outcome: ceiling`»,
// which is what a consumer of THIS FILE would have built against — and both acceptance verifiers
// found it independently, which is the strongest signal a wrong sentence can get. The engine attaches
// it on FIVE terminal branches (pipeline/events.go), because a ceiling no longer cancels its siblings
// and a run can therefore latch on money and still depart as something else:
//
// - `ceiling` — the tidy case;
// - `failed` — an infra failure or a crash landed on top of a caught ceiling;
// - `stopped` — a caught SIGTERM after the money had already run out;
// - `bank_stop` — a signing stop on a book that had touched its ceiling.
//
// In every one of those the buyer's book is in the SAME state, so a ledger attached only to the tidy
// exit would be missing from exactly the messy ones a reader needs it for.
//
// ⚠ AND THE CONVERSE IS FALSE TOO, so do not build on it either: `outcome: ceiling` does NOT imply
// this field. It is absent when the run stopped before it learned the book's cut, because the
// counters it is made of are seeded from that cut and there is nothing honest to report without one.
//
// So: PRESENT means «a spend ceiling was reached in this run, and here is what it bought». ABSENT
// means «no ceiling, or no counters» — never «nothing was bought».
//
// ⛔ IT IS NOT `Volume` UNDER ANOTHER NAME, and the two must not be merged. A volume grant is decided
// BEFORE the waves and its seven counters are a PLAN trued up afterwards; a spend ceiling stops the
// run in the middle of work the plan said would happen, so those counters describe units that were
// admitted and never done — Delivered over-counts, LeftFresh under-counts, and a run that stopped
// short would report «reached the end of the book». The plan has no bucket for «admitted, not
// finished», so a money stop is given its own ledger, counted from what the run RESOLVED rather than
// from what it intended.
//
// The counters come from the same per-wave tally `progress` is published from, so the running channel
// and the terminal one can never quote two different numbers about one run.
Money *MoneyLedger `json:"money,omitempty"`
}
// MoneyLedger is what a purchase bought before the money ran out, in OUTPUT UNITS — the granularity
// every read model counts in.
//
// It answers the two questions a buyer of a stopped book has, and nothing else: what do I have, and what
// is still owed. It carries no prices: what a stage or a call costs stays inside the engine (ПТ-33),
// and the one figure that leaves is the shortfall on the `ceiling` event beside it.
type MoneyLedger struct {
// UnitsResolved is output units that have REACHED A VERDICT in their shipping wave, including units
// earlier runs resolved (the counters are seeded from the store, so a resumed run reports the book's
// state and not its own slice of it).
//
// ⚠ RESOLVED, NOT DELIVERED, AND THE WORD IS THE WHOLE OF THE FIELD'S HONESTY. A unit that resolved
// FLAGGED is counted here and it may carry no readable text at all — it went through the wave and
// came out needing a human. Calling that «delivered» is the exact lie the volume ledger was split in
// two to stop telling (VolumeLedger's Delivered/Flagged), and it is a lie a buyer would act on. What
// a reader wants delivered counts for lives in the unit stream, which says of every unit whether it
// shipped.
UnitsResolved int `json:"units_resolved"`
// UnitsDeferred is output units the book still owes: in the cut and not resolved. Zero with a ceiling
// stop is a real state and not a contradiction — the last unit can be resolved by the very call that
// exhausted the ceiling.
UnitsDeferred int `json:"units_deferred"`
}
// VolumeLedger is what a purchase of N output units actually bought, in the unit the seam sells in.
//
// The split is the whole value. «Paid» alone cannot tell a reader whether the money became book: a unit
// can be paid for and come back FLAGGED with nothing shippable, and a unit can be paid for a second time
// because its snapshot moved — real work and a real charge, but not new book. The remainders split the
// same way, and only ONE of them may ever be offered for sale: LeftFresh is book nobody has, LeftRework is
// book the reader already has that is merely unrefreshed.
type VolumeLedger struct {
// MaxUnits is the grant that was in force.
MaxUnits int `json:"max_units"`
// Delivered — paying units that had never been completed before. This is what «buy ten chapters» means,
// and it is the count that can EXCEED max_units: a run also finishes units an earlier run started and
// never shipped, and those are not charged against this grant (backend volume.go, backlog row 232). The
// engine knows how many of them there were; the frame does not carry that split yet, so a reader seeing
// delivered > max_units is seeing interrupted book being finished, not a ceiling that failed to hold.
Delivered int `json:"delivered"`
// Reworked — paying units that were already complete and are being made again under a moved snapshot.
Reworked int `json:"reworked"`
// Flagged — paid units that resolved flagged and shipped no text. Money spent, no chapter produced.
// It is filled from what HAPPENED, not from the plan, so Delivered+Reworked+Flagged is what was paid.
Flagged int `json:"flagged"`
// Free — units that rode along at $0 (resumed or re-pinned). One of the two reasons a run touches more
// units than it was granted; the other is the interrupted book counted inside Delivered above.
Free int `json:"free"`
// LeftFresh — undelivered units still in the book. The ONLY remainder anybody may be invited to buy.
LeftFresh int `json:"left_fresh"`
// LeftRework — completed units still carrying a superseded snapshot. Unrefreshed, not unbought;
// presenting these as stock for sale is how re-payment becomes a product.
LeftRework int `json:"left_rework"`
}
// The outcomes this engine writes. They mirror the shell contract of cmd/tmctl (0/2/3/4/5/1) so a reader
// never has to do exit-code archaeology over a stream that ended cleanly.
const (
OutcomeClean = "clean"
OutcomeFlagged = "flagged"
OutcomeBankStop = "bank_stop"
OutcomeCeiling = "ceiling"
OutcomeStopped = "stopped"
OutcomeFailed = "failed"
)
// The waves a unit can be resolved by.
const (
WaveDraft = "draft"
WaveEdit = "edit"
)
// The ceiling scopes.
const (
ScopeBook = "book"
ScopeDay = "day"
)
// Line renders one journal line — the envelope, WITHOUT its newline. The bytes it returns are the line:
// they are stored verbatim and re-projected verbatim, because a reader compares a re-read line against
// the sha256 it recorded and a re-render with a fresher timestamp would read as a payload conflict
// (tail.go: ErrPayloadConflict) and quarantine the projection.
func Line(seq int64, t Type, at time.Time, data any) ([]byte, error) {
payload, err := json.Marshal(data)
if err != nil {
return nil, fmt.Errorf("runevents: marshal %s payload: %w", t, err)
}
line, err := json.Marshal(Envelope{Seq: seq, Type: t, Time: at.UTC(), Data: payload})
if err != nil {
return nil, fmt.Errorf("runevents: marshal %s envelope: %w", t, err)
}
return line, nil
}
// MicroUSD converts a ledger figure to the integer micro-USD the seam carries, rounding UP: the ledger
// is a lower bound on what a provider billed (a 2xx whose body did not decode is settled at its
// estimate), so the seam must never round the shortfall away. A negative figure cannot exist in the
// ledger and is clamped rather than sign-extended into a nonsense counter.
func MicroUSD(usd float64) int64 {
if !(usd > 0) { // also catches NaN
return 0
}
return int64(math.Ceil(usd*1e6 - 1e-6))
}