package ingest import ( "encoding/json" "fmt" ) // bank.go: the ALLOWLISTED reading of the engine's whole-bank sidecar (`.bank.json`, // engine backlog row 125, landed D39.122). // // The bank lives in the engine's private SQLite, which this side must never open (D39.85), so the // sidecar is the only channel there is. It is written atomically (temp + rename), which is what makes // it readable while a run is going — the one engine artifact of the three that does not have to wait // for a boundary. // // ⚠ Every vocabulary crossing here is TRANSLATED and none is passed through. `ruby` is Japanese // furigana — pair-specific data in a shared layer — `mined` is the name of a pipeline stage and // `draft` the name of a wave; the contract renamed all six values in 0.3.0 precisely so that none of // them reaches a client (canon §TermStatus/§TermOrigin, companion §2.8). Doing the mapping at the // seam rather than at projection time is deliberate: what is STORED is then already what a client // reads, and no later path can leak a word by forgetting to translate it. // BankTerm is one row of the bank, in the contract's own words. type BankTerm struct { // ID is the engine's derived key over the row's uniqueness key (src, sense, window). Stable // across the bank being rebuilt — which it is on every run — and NOT across the book being cut // differently, because the window is in chapter numbers. ID string Src string Dst string Kind string // "" when the engine did not decide: a legal state the row still needs signing in Status string // proposed | in_progress | approved Origin string // given | annotated | found Sense string // SinceChapter / UntilChapter are nil for "no boundary". The engine's own sentinel is 0, which is // a value chapter numbering cannot produce (it starts at 1) and which meant two different things // in two fields. SinceChapter *int UntilChapter *int } // Bank is the whole read-out. type Bank struct { // Boundary is which of the run's five publishing moments wrote this document, in the contract's // words. It is read WITH the sections and not instead of them: the engine publishes `offered` at // the signing stop and nowhere else, so a section standing at any other boundary is a document // this build does not understand, and serving its rows would invent a stop that is not there. Boundary string // RunID is the anchor's other half — the id the engine announces its event stream under, which is // the id this platform minted for the attempt (pgstore.EngineStreamID). Empty means the write had // no run identity: unknown, never "current". Read and stored, never served: it is an engine // identity, and the seam does not export those. RunID string Terms []BankTerm // Offered is what the signing stop is asking about — mined rows NOT in the bank yet, in the stop's // own ranking. A pointer to the slice because nil ("the document carried no section") and an empty // slice ("the stop asked nothing") are different answers to a person looking at an empty screen, // and a plain slice decodes both to nil. Offered *[]OfferedTerm // Consolidation is how complete the bank being signed is, nil when nothing measured it. The engine // makes the section absent rather than zeroed where the paid pass never ran, because a zeroed one // answers "consolidated 0, unanswered 0" — which reads as "nothing is missing". Consolidation *Consolidation } // OfferedTerm is one row of a signing stop's table, in the contract's own words. // // ⛔ The numbers are pointers and the strings are not. For a field the engine writes ALWAYS, absence // means a build without that field wrote the document — and for these four the zero value is itself a // measurement (`freq: 0` is the engine's "only the draft side saw it"; a stated `conf: 0` is the row a // signer must look at), so decoding absence to zero publishes a figure nobody took. Measured, not // hypothetical: `conventions` is absent from all 69 rows of one bought stop read-out and present on // all 66 of the other, because the field was added between the two runs. The empty string carries no // such second meaning, so the strings lose nothing by letting absence land on them. type OfferedTerm struct { Src string // Dst is the consolidated rendering, "" when this read-out carries none — and it does NOT say why. // The engine's own sheet spends four facts on that glyph, one of which ("the bank already renders // this surface, nothing to decide") it distinguishes as BankStopRow.SettledByBank and deliberately // does not project here. Nothing downstream may read "" as "the service could not translate it". Dst string // Kind and Channel are "" for what this build cannot name — a kind the engine did not decide, a // detector outside the closed set. Never a guess: Channel is how a person judges how well // corroborated a row is, and invented corroboration is what reading further cannot undo. Kind, Channel string // Conventions is NOT re-derived from len(Variants) when absent, although the engine's projection // makes them equal today: deriving is this side re-implementing engine law, and the engine's own // comment records that reading one of these numbers as the other already cost two sessions a // misfiled defect. Freq, Spread, Conventions *int // Confidence is nil when the service stated none — INCLUDING the engine's own spelling for it, a // negative number. Folded here at the seam rather than at projection time, like every other // crossing in this file and for the reason the note at the top gives: a translation left to a // later path is one a later path can forget, and this one WAS forgotten — the canon promises // `0..100` or null and a build that carried the minus through would have shown a person "-1 %". // // ⚠ It folds "the read-out carried no `conf` at all" into the same nil, and that is measured // rather than assumed: `conf` entered the projection in the SAME commit as the section it lives // in, so no build emitting the section omits it (0 of 135 rows across two bought read-outs). // THE CONDITION THAT ENDS THAT: a build that starts omitting it — then absence stops being // unreachable and the two causes need telling apart again. Confidence *int // Invented — the rendering is not one the drafts proposed, the class the engine names as the one to // read first. A plain bool because the engine omits the field when false. ⚠ THE CONDITION THAT ENDS // THAT: the day `omitempty` leaves the engine's field, absence stops meaning false and this has to // become a pointer. Nothing here goes red on that day. Invented bool // Two lists and never one: "this run disagreed with itself" and "the book already calls it // something else" are different decisions. Contradicts, BankHolds []string // The drafts' renderings, best-ranked first, in the engine's own labelling — opaque here and never // parsed. It keeps a reader beside that writer precisely because a re-derived parser "would keep // working until the day the label gains a field, and then it would report numbers rather than an // error". Variants []string } // Consolidation is what the paid terminology pass MANAGED. Six numbers and not one, because a cut // RENDER pass leaves the bank partial while a cut CLASSIFY pass leaves it whole with the types // unrefined — a different budget and a different remedy — and rows the role was never asked about are // a saving rather than a gap. type Consolidation struct { // Complete comes from the engine ready-made: it is computed off the render pass alone, and a // second mechanism answering that here is how the two come to disagree. Complete bool // Two cuts and never one sum: an operator raises one budget or the other, never "the" budget. RenderBatchesDropped, ClassifyBatchesDropped int Consolidated, Declined int // Unanswered conflates the role's silence with the budget's cut — it means "the role stayed // silent" only where Complete is true. Unanswered int // NeverAsked is the opposite of a gap: the bank already renders the surface and every draft agreed. NeverAsked int } // The contract's TermStatus and TermOrigin values. const ( TermProposed = "proposed" TermInProgress = "in_progress" TermApproved = "approved" OriginGiven = "given" OriginAnnotated = "annotated" OriginFound = "found" ) // The contract's ReadOutBoundary values, stored but never served. The engine's own names are pipeline // vocabulary (`bank-mining` is a stage, `redrive` an operator's verb), which the note at the top of // this file refuses to pass through. // // ⛔ BoundaryUnknown covers both a name this build does not know and the empty one a document from // before the anchor carries. Safe to collapse because the consequence is identical — nothing says this // is a stop — and the one thing neither may become is BoundarySignatureRequested. const ( BoundaryRunStarted = "run_started" BoundaryTermsTakenUnsigned = "terms_taken_unsigned" BoundarySignatureRequested = "signature_requested" BoundaryRunFinished = "run_finished" BoundaryBookReseeded = "book_reseeded" BoundaryUnknown = "unknown" ) // The contract's OfferedTermChannel values. The engine's four — `mined`, `banknote`, `both`, `alias` — // are pipeline vocabulary, one of them the very word the contract renamed in 0.3.0 so it could not // reach a client. const ( ChannelSourceText = "source_text" ChannelTranslatedText = "translated_text" ChannelBoth = "both" ChannelAlias = "alias" ) type wireBank struct { Version string `json:"bank_version"` BookID string `json:"book_id"` // Additive on tm-bank-v1 and absent in documents written before the anchor existed, so an absent // AsOf lands on BoundaryUnknown rather than refusing the document: such a read-out is old, not // malformed. AsOf string `json:"as_of"` RunID string `json:"run_id"` Terms []struct { ID string `json:"id"` Src string `json:"src"` Dst string `json:"dst"` Kind string `json:"kind"` Status string `json:"status"` Origin string `json:"origin"` Sense string `json:"sense"` SinceChapter int `json:"since_chapter"` UntilChapter int `json:"until_chapter"` } `json:"terms"` // Pointers, so that an absent member stays distinguishable from an empty one — see Bank.Offered. Proposed *[]wireOffered `json:"proposed"` Consolidation *wireConsolidation `json:"consolidation"` } type wireOffered struct { Src string `json:"src"` Dst string `json:"dst"` Kind string `json:"kind"` Channel string `json:"channel"` Freq *int `json:"freq"` Spread *int `json:"spread"` Conventions *int `json:"conventions"` Conf *int `json:"conf"` Invented bool `json:"invented"` Contradicts []string `json:"contradicts"` BankHolds []string `json:"bank_holds"` Variants []string `json:"variants"` } type wireConsolidation struct { Complete bool `json:"complete"` RenderBatchesDropped int `json:"render_batches_dropped"` ClassifyBatchesDropped int `json:"classify_batches_dropped"` Consolidated int `json:"consolidated"` Declined int `json:"declined"` Unanswered int `json:"unanswered"` NeverAsked int `json:"never_asked"` } // DecodeBank parses a bank read-out, and refuses anything that does not identify itself as one — the // same guard the manifest carries, for the same reason: an empty bank and a document this build // cannot read decode identically, and one of them would replace a book's whole bank with nothing. func DecodeBank(b []byte) (Bank, error) { var doc wireBank if err := json.Unmarshal(b, &doc); err != nil { return Bank{}, fmt.Errorf("ingest: decode bank: %w", err) } if doc.Version == "" { return Bank{}, fmt.Errorf("ingest: decode bank: the document carries no bank_version, so it is not a bank") } out := Bank{ Boundary: readOutBoundary(doc.AsOf), RunID: doc.RunID, Terms: make([]BankTerm, 0, len(doc.Terms)), } for _, t := range doc.Terms { status, ok := termStatus(t.Status) if !ok { // A status this build has never heard of. Dropping the row is the wrong answer — it would // silently shrink a bank somebody has to sign — and guessing `approved` would carry an // unsigned term into the book as canon, so it is read as the state that asks for a // decision. status = TermProposed } origin, ok := termOrigin(t.Origin) if !ok { // Provenance is what the person signing judges trust by, and there is no safe guess: a row // whose origin this build cannot name is reported as the one that claims the least about // where it came from. origin = OriginFound } out.Terms = append(out.Terms, BankTerm{ ID: t.ID, Src: t.Src, Dst: t.Dst, Kind: termKind(t.Kind), Status: status, Origin: origin, Sense: t.Sense, SinceChapter: chapterBound(t.SinceChapter), UntilChapter: chapterBound(t.UntilChapter), }) } out.Offered = decodeOffered(doc.Proposed) out.Consolidation = decodeConsolidation(doc.Consolidation) return out, nil } // decodeOffered reads the stop's section. The nil check IS the section's contract: no member means no // section, a section with no rows means a stop that asked nothing. func decodeOffered(rows *[]wireOffered) *[]OfferedTerm { if rows == nil { return nil } out := make([]OfferedTerm, 0, len(*rows)) for _, r := range *rows { out = append(out, OfferedTerm{ Src: r.Src, Dst: r.Dst, Kind: termKind(r.Kind), Channel: offeredChannel(r.Channel), Freq: statedCount(r.Freq), Spread: statedCount(r.Spread), Conventions: statedCount(r.Conventions), Confidence: statedConfidence(r.Conf), Invented: r.Invented, Contradicts: r.Contradicts, BankHolds: r.BankHolds, Variants: r.Variants, }) } return &out } // decodeConsolidation copies the numbers and derives none of them. // // ⚠ A section that is PRESENT and empty decodes to "not complete, nothing consolidated" — the // cautious direction, and the opposite of the rule for the numbers above, where a zero would have // invented a measurement. The two look contradictory until you ask which way each zero errs. // // ⛔ AND THE CONDITION UNDER WHICH THAT STOPS HOLDING, because it holds only by an accident of how // the writer grew: the seven fields entered the engine's projection in ONE commit, so no build emits // a partial section and "present" always means all seven. The day the engine adds an eighth number, // a document written by the older build decodes it as zero — and `unanswered: 0` says "the role // stayed silent about nothing", which is the false zero this whole reader exists to refuse. It is // cautious for `complete` and a lie for the counters, and the difference appears the moment the // field set moves. Then these become pointers, like the four above. func decodeConsolidation(c *wireConsolidation) *Consolidation { if c == nil { return nil } return &Consolidation{ Complete: c.Complete, Consolidated: c.Consolidated, Declined: c.Declined, Unanswered: c.Unanswered, NeverAsked: c.NeverAsked, RenderBatchesDropped: c.RenderBatchesDropped, ClassifyBatchesDropped: c.ClassifyBatchesDropped, } } // BoundaryOrUnknown is the closed set as a GUARD, for a caller that did not come through DecodeBank. // The zero value of a Bank carries no boundary, and a read-out whose moment nobody named is exactly // what BoundaryUnknown is for — refusing instead would fail a whole bank save over a label. func BoundaryOrUnknown(s string) string { switch s { case BoundaryRunStarted, BoundaryTermsTakenUnsigned, BoundarySignatureRequested, BoundaryRunFinished, BoundaryBookReseeded: return s } return BoundaryUnknown } // KindOrNone and ChannelOrNone are the same guard for the two closed vocabularies an offered term // carries, whose "cannot name it" is "" rather than a value. // // ⛔ They are NOT what `nullif($, ”)` does in the statement, and the difference is a live failure // rather than a nicety: `nullif` turns the EMPTY value into null, while these turn an UNKNOWN // non-empty one into the empty value first. Measured — an engine word that leaked this far without a // guard does not land as "not named", it violates the column's CHECK and takes the WHOLE bank save // down, at the one boundary a person is standing at. func KindOrNone(s string) string { return termKind(s) } func ChannelOrNone(s string) string { switch s { case ChannelSourceText, ChannelTranslatedText, ChannelBoth, ChannelAlias: return s } return "" } // ⛔ A NUMBER OUTSIDE THE RANGE THE CONTRACT DECLARES IS READ AS «NOT STATED» — the same discipline // the vocabularies above follow, one floor down: a value this build cannot name becomes the one that // claims the least, never a guess and never a pass-through. // // It is a CLASS and not one field, which is how it was found: the engine spells "the reply carried no // confidence" as a negative, and a build that let that through would show a person "-1 %" on the one // screen that asks them to decide. But the canon bounds `freq`, `spread` and `conventions` too, and // nothing there is a sentinel — a negative count is simply a document this build does not understand, // and publishing it would be the same lie in a quieter place. // // ⚠ The bounds are mirrored from the canon rather than read from it at runtime, like the closed // vocabularies above, and like them they are gated against it: // TestEveryRangeTheCanonDeclaresIsOneThisReaderEnforces reads `minimum`/`maximum` out of the canon and // fails on any ranged member this file does not bound. THE CONDITION THAT ENDS THE MIRRORING: a canon // that gives a member a range this file has no branch for — which is exactly what that gate says. func statedCount(v *int) *int { return inRange(v, 0, -1) } func statedConfidence(v *int) *int { return inRange(v, 0, 100) } // inRange keeps a value the contract would accept and drops one it would not. A negative max means // "no ceiling", which is what the canon says by declaring `minimum` alone. func inRange(v *int, min, max int) *int { if v == nil || *v < min || (max >= 0 && *v > max) { return nil } return v } func readOutBoundary(engine string) string { switch engine { case "run-start/seeded": return BoundaryRunStarted case "bank-mining/auto-continue": return BoundaryTermsTakenUnsigned case "bank-mining/signature-stop": return BoundarySignatureRequested case "run-finished": return BoundaryRunFinished case "redrive/re-seeded": return BoundaryBookReseeded } return BoundaryUnknown } // offeredChannel returns "" for a detector outside the closed set — the same spelling termKind uses // for a value this build cannot name, so one convention carries "not nameable" all the way out. func offeredChannel(engine string) string { switch engine { case "mined": return ChannelSourceText case "banknote": return ChannelTranslatedText case "both": return ChannelBoth case "alias": return ChannelAlias } return "" } func termStatus(engine string) (string, bool) { switch engine { case "auto": return TermProposed, true case "draft": return TermInProgress, true case "approved": return TermApproved, true } return "", false } func termOrigin(engine string) (string, bool) { switch engine { case "seed": return OriginGiven, true case "ruby": return OriginAnnotated, true case "mined": return OriginFound, true } return "", false } // termKind passes the engine's classification through the contract's closed vocabulary. A kind // outside it — including the empty one the engine legitimately produces for a candidate it could not // classify — becomes "kind not decided", which is a state the contract has and a client must render. // Passing an unknown value through instead would hand a generated client a value its union does not // contain. func termKind(engine string) string { switch engine { case "name", "place", "title", "term", "nickname": return engine } return "" } func chapterBound(n int) *int { if n <= 0 { return nil } return &n }