textmachine/backend/internal/chunk/chunker.go

// Package chunk turns a source file into the ordered units of translation work: ingest decodes the
// book (txt/epub, with encoding detection and ruby capture) into per-chapter normalized text, and
// the chunker segments that text into draft chunks and groups them into edit units.
//
// Both halves are deterministic and versioned: identical input yields identical chunk boundaries,
// because those boundaries decide what goes on the wire — a shifted boundary is a re-translation of
// a book already paid for. All language-specific structure (chapter markers, CJK section numerals,
// heading rules, sentence terminators, abbreviations) arrives as DATA via *lang.Pack and the
// embedded language data, so a new pair needs no code here.
package chunk

import (
	"strconv"
	"strings"
	"unicode"

	"textmachine/backend/internal/lang"
	"textmachine/backend/internal/text"
)

// chunker.go: the step-3a SOURCE segmenter. It turns the per-chapter normalized
// text produced by ingest.go into an ordered list of (chapter, chunk) units for
// the runner loop. WS2 (layer 1) moved the packing budget to OUTPUT (ru) tokens via
// the fertility estimate (est_out = f_cjk·cjk + f_other·other over source char-classes),
// decoupling draft sizing (small chunk, COGS/coverage/alignment) from the EDIT unit
// (coarse chapter-scale unit for cross-chunk cohesion): pack whole paragraphs to
// DraftBudgetOut, and when a single paragraph exceeds it descend to SENTENCE boundaries
// (never splitting a sentence; a lone over-budget sentence is a passthrough
// OversizedSentence chunk). Edit units then group WHOLE draft chunks to EditCeilingOut.
//
// Contract kept intact for the loop (bookrun.go/chunkrun.go depend on it): SplitChunks emits
// Chunk{Chapter,ChunkIdx,Text,…}; ChunkIdx resets to 0 per chapter; the new fields (EstOut,
// OversizedSentence, EditUnitID) are ADDITIVE; the function is a PURE, deterministic function of
// (chapters, SegBudget) (no time/rand, no map iteration). Its behavior is versioned by
// chunkerVersion (render.go); the SegBudget is snapshot-folded via segmentationSnap — so changing
// the segmentation OR the budget/fertility is a loud --resnapshot re-translation, never a silent
// cache miss (§3.4/R6/D5.2/D30.9).
//
// IMPORTANT: this sentence splitter is a SEPARATE, independent piece of logic from
// the coverage gate's splitSentences (internal/checks). That one is a bit-for-bit port of the Python oracle
// (refusal_bench.py) over the RUSSIAN OUTPUT and is Python-locked (D12 Q1). THIS one
// segments the SOURCE (zh/ja/en), is versioned by chunkerVersion (Р2 = code), and is
// free to be more faithful (closing-bracket absorption, an abbreviation guard). Do
// not couple them.

// Chunk is one ordered unit of translation work.
type Chunk struct {
	Chapter  int // 1-based
	ChunkIdx int // 0-based within its chapter
	Text     string
	// EstOut is the fertility-estimated OUTPUT (ru) token cost of this chunk (WS2, layer 1):
	// est_out = fert.CJK·cjk_src + fert.Other·other_src over the source char-classes. It is the
	// unit the draft budget packs against (DraftBudgetOut), decoupling draft sizing from the
	// old input-token heuristic (L2-budget-wrong-unit). Additive field; the {Chapter,ChunkIdx,Text}
	// contract the loop/ingest/status depend on is untouched.
	EstOut float64
	// OversizedSentence marks a chunk that is a SINGLE source sentence whose est_out alone exceeds
	// DraftBudgetOut (WS2 §2б). It is passthrough (never clause-split — clause splitting is
	// boundary machinery, anti-scope), and the flag makes the un-budgetable unit OBSERVABLE
	// ("degradation is not silent") rather than silently oversized.
	OversizedSentence bool
	// Heading is the deterministic chapter title, non-empty ONLY on a chapter's FIRST chunk (ChunkIdx 0)
	// when the chapter opened with a source structural header (RECOGNISED by the source language's
	// lang.SourceStructure) AND the pair supplies a template. The chunker STRIPS the source marker from ch.Text (so the model never
	// renders its own «Раздел 2»/«Первая глава»/an orphaned « :») and renders this from the rule's template
	// («Глава N»); the read-models PREPEND it to the chapter's first output unit. "" for every non-first
	// chunk, every chapter without a header, and every book without a heading rule (in which case the source
	// header stays in ch.Text and chunking is byte-identical to before the feature existed).
	Heading string
	// EditUnitID is the 0-based book-global id of the coarse EDIT unit this draft chunk belongs to
	// (WS2 decoupling: draft = small chunk, edit = large unit). An edit unit is a greedy grouping of
	// WHOLE draft chunks up to EditCeilingOut (per chapter), so the edit wave's editor reads a unit's draft as
	// the concatenation of its member chunks' the draft wave outputs — clean reconstruction by construction (no
	// straddle; verified: grouping whole chunks reproduces the paragraph-packed count 37 with 0
	// straddle on the rerun corpus). Draft chunks never cross an edit-unit boundary.
	EditUnitID int
}

// SegBudget is the resolved output-token segmentation budget (WS2 §2в): the draft-chunk and
// edit-unit ceilings in ru-OUTPUT tokens, plus the per-pair fertility coefficients that convert
// source char-classes to an output-token estimate. Resolved from config.Segmentation by the runner
// and snapshot-folded via segmentationSnap, so a budget/fertility edit is a loud --resnapshot
// (D30.9). The classifier (TokenClassCounts) uses the REAL unicode.RangeTable of EstimateTokens
// (Han|Hiragana|Katakana|Hangul) for generality (§0.1) — NOT script-specific ord ranges.
type SegBudget struct {
	DraftBudgetOut float64 // draft-chunk ceiling, ru-output tokens (default 1797)
	EditCeilingOut float64 // edit-unit ceiling, ru-output tokens (default 3200, gated arm at 8000)
	FertCJK        float64 // output tokens per CJK source char (default 1.1978)
	FertOther      float64 // output tokens per non-CJK/non-space source char (default 0.3852)
}

// EstOut applies the fertility formula to pre-counted source char-classes — identical to
// est_out over the joined chunk text (whitespace is class-free, as TokenClassCounts drops it).
func (s SegBudget) EstOut(cjk, other int) float64 {
	return s.FertCJK*float64(cjk) + s.FertOther*float64(other)
}

// SplitChunks segments the ordered, per-chapter NORMALIZED text of a Document
// (ingest.go already applied text.NormalizeSource and split chapters) into an ordered
// chunk list. Chapters that yield no text (blank/whitespace-only, e.g. an epub
// cover/nav page) do NOT consume a chapter number, so numbering stays dense over
// real content (Phase-0 backward compat). Fully deterministic. Each chunk carries its
// EstOut + OversizedSentence flag + EditUnitID (WS2); the edit-unit id is monotone across
// the book so a the edit wave unit is uniquely addressable.
func SplitChunks(chapters []string, seg SegBudget, rule *lang.ChapterRule, abbrevs map[string]bool) []Chunk {
	chunks, _, _ := SplitChunksWithChapters(chapters, seg, rule, abbrevs)
	return chunks
}

// SplitChunksWithChapters is SplitChunks plus the INGESTED text of every chapter that consumed a number,
// index-aligned to the chapter numbers it emitted (kept[0] is chapter 1). It exists so a caller that has
// to identify a chapter — the persisted manifest, backlog row 100 — gets the chapter's own text from the
// ONE place that knows which chapters consume a number, instead of re-deriving that rule and drifting
// from it the first time an "empty chapter" edge case changes.
//
// The text returned is the chapter AS INGESTED — before stripHeading. That is deliberate: an identity
// built on it survives a heading-rule edit (a data change in the pair pack) and a chunker/budget change,
// because neither touches what ingest produced. Nothing else here changes: SplitChunks is this function
// with the extra results dropped, so every existing caller is byte-identical.
//
// keptIdx gives each kept chapter's index in the INPUT slice. It exists so a caller holding data parallel to
// the input — the per-chapter source titles — can select the same chapters WITHOUT re-deriving the
// "an empty chapter takes no number" rule. Re-deriving it is how a title ends up one chapter late, and that
// error is invisible: every chapter still has A title.
func SplitChunksWithChapters(chapters []string, seg SegBudget, rule *lang.ChapterRule, abbrevs map[string]bool) (chunks []Chunk, kept []string, keptIdx []int) {
	var out []Chunk
	chapterNo := 0
	editUnitID := 0
	for idx, ingested := range chapters {
		// Title policy (pack-13): detect a leading structural header, render it deterministically and
		// STRIP its marker from the text the model sees. An absent TEMPLATE (no pack / no heading.txt) or a chapter
		// with no header is a NO-OP — headingText is "" and chapText is unchanged, so a book that does not
		// opt in produces byte-identical chunks. Runs BEFORE splitParagraphs so the stripped subtitle is
		// re-paragraphed normally.
		headingText, chapText := stripHeading(ingested, rule)
		paras := splitParagraphs(chapText)
		if len(paras) == 0 {
			continue // an empty chapter does not consume a chapter number
		}
		chapterNo++
		kept = append(kept, ingested)
		keptIdx = append(keptIdx, idx)
		chapterChunks := chapterDraftChunks(chapterNo, paras, seg, abbrevs)
		if headingText != "" && len(chapterChunks) > 0 {
			chapterChunks[0].Heading = headingText // the chapter's first chunk carries the deterministic title
		}
		assignEditUnits(chapterChunks, seg, &editUnitID)
		out = append(out, chapterChunks...)
	}
	return out, kept, keptIdx
}

// stripHeading detects a chapter-leading structural header and returns the DETERMINISTIC rendered title
// plus the chapter text with the source marker removed (pack-13 title policy). It reads a ChapterRule: the
// SOURCE grammar recognises the header, the PAIR template renders it.
//
// The no-op condition is an ABSENT TEMPLATE, not a nil rule — the runner always builds a rule, because every
// book has a source language even when it has no pair pack. No template, no header, or no grammar: it
// returns ("", chapter) unchanged and the chunks are byte-identical to a run without the feature. The
// marker+numeral+unit prefix is stripped and the SUBTITLE (if any) is kept as ordinary body — so «第一节：
// 纵身亡魔心仍不悔» yields ("Глава 1", "纵身亡魔心仍不悔\n…"): the reader gets a uniform «Глава N» PLUS the
// translated subtitle, and the model never renders the chapter number. Only the FIRST line is inspected
// (a header is a chapter's opening line; a chapter with a preamble before its header is a rare edge left
// to the pre-pack behaviour, noted in the pack report). Deterministic and pure.
func stripHeading(chapter string, cr *lang.ChapterRule) (heading, stripped string) {
	if cr == nil || cr.Template == "" {
		return "", chapter
	}
	parts := strings.SplitN(chapter, "\n", 2)
	n, subtitle, ok := matchHeaderLine(parts[0], cr)
	if !ok {
		return "", chapter
	}
	heading = strings.ReplaceAll(cr.Template, "{n}", strconv.Itoa(n))
	rest := ""
	if len(parts) == 2 {
		rest = parts[1]
	}
	switch {
	case subtitle == "":
		return heading, strings.TrimLeft(rest, "\n") // the whole header line + its trailing blank go away
	case rest == "":
		return heading, subtitle
	default:
		return heading, subtitle + "\n" + rest
	}
}

// ApplyHeading prepends a chapter's DETERMINISTIC title to its first output unit's final text (pack-13
// title policy). It is a PURE, deterministic projection applied at assembly time (waverun outcome +
// export), NEVER stored in a checkpoint — so a resume re-derives it for free from the manifest re-chunk,
// and a book with no heading rule (heading=="") is byte-identical. It prepends ONLY to a NON-EMPTY final
// text: a fully-flagged/empty unit keeps "" (the "not translated" export semantics), and the title
// reappears when the unit is redriven ok. The separator is a blank line so «Глава N» reads as a heading
// above the (subtitle + body) prose.
func ApplyHeading(heading, finalText string) string {
	if heading == "" || finalText == "" {
		return finalText
	}
	return heading + "\n\n" + finalText
}

// matchHeaderLine reports whether a single line is a structural header under the rule and returns the
// parsed section number + the trimmed subtitle after the marker. Shape: <marker><numeral-run><unit-rune>,
// then EITHER end-of-line OR a SEPARATOR (not a content glyph) — mirroring ingest.isChapterHeader's
// precision guard so «第一回见面» (回 a measure word glued to the content 见) is NOT a header, while
// «第一节：…» / «第1章 …» is. Leading whitespace is tolerated; the subtitle is the remainder with any
// leading separators (：: 、,.。- — 　·) trimmed. Deterministic.
func matchHeaderLine(line string, cr *lang.ChapterRule) (n int, subtitle string, ok bool) {
	t := strings.TrimSpace(line)
	st := cr.Structure
	afterMarker, ok := st.MatchMarker(t)
	if !ok {
		return 0, "", false
	}
	rs := []rune(afterMarker)
	// Tolerate whitespace between the marker and the number («Chapter 12», «Глава 3»). A CJK header («第12章»)
	// has none, so the skip is additive.
	j := 0
	for j < len(rs) && unicode.IsSpace(rs[j]) {
		j++
	}
	i := j
	for i < len(rs) && isHeadingNumeral(rs[i]) {
		i++
	}
	if i == j { // need at least one numeral
		return 0, "", false
	}
	num, parsed := parseSectionNumeral(string(rs[j:i]))
	if !parsed {
		return 0, "", false
	}
	after := rs[i:]
	if len(st.Units) > 0 {
		// A unit-bearing rule (CJK 章/节/…) REQUIRES the unit rune right after the number — the guard that
		// keeps «第一次» (a measure word) from reading as a header.
		if len(after) == 0 || !st.Units[after[0]] {
			return 0, "", false
		}
		after = after[1:]
	}
	if len(after) > 0 && isHeaderContentRune(after[0]) {
		return 0, "", false // a content glyph glued to the number/unit → a measure/ordinal in prose, not a header
	}
	return num, strings.TrimSpace(strings.TrimLeftFunc(string(after), isHeadingSeparator)), true
}

// isHeadingNumeral reports whether a rune can be part of a chapter-number run: an Arabic digit
// (half/fullwidth) or a CJK numeral character (the shared lang.CJKSection — pair-14 §4, the SAME source
// ingest's chapter-numeral regex reads, so the two can never byte-drift apart).
func isHeadingNumeral(r rune) bool {
	switch {
	case r >= '0' && r <= '9', r >= '０' && r <= '９':
		return true
	}
	return lang.DefaultCJKSection().IsHeadingNumeral(r)
}

// isHeaderContentRune reports whether a rune is CONTENT (a letter/digit/ideograph/kana) rather than a
// separator — the guard that keeps a glued measure word from reading as a header (ingest parity). It is the
// single source of truth for the content/separator split: ingest.isHeaderSeparator is its exact complement.
func isHeaderContentRune(r rune) bool {
	return unicode.IsLetter(r) || unicode.IsDigit(r) ||
		unicode.Is(unicode.Han, r) || unicode.Is(unicode.Hiragana, r) || unicode.Is(unicode.Katakana, r)
}

// isHeadingSeparator reports whether a rune separates the marker from the subtitle (trimmed off the head
// of the subtitle): CJK/ASCII colons, commas, periods, dashes, the ideographic space and middle dots. This
// is a DISTINCT, narrower whitelist — NOT the complement of isHeaderContentRune (that is isHeaderSeparator).
func isHeadingSeparator(r rune) bool {
	switch r {
	case '：', ':', '、', '，', ',', '.', '。', '-', '—', '–', ' ', '\t', '　', '·', '・':
		return true
	}
	return false
}

// parseSectionNumeral parses a chapter-number run into an int (Arabic half/fullwidth OR standard CJK
// numerals up to the thousands — well past any chapter count). A digit accumulates positionally; a small
// unit (十百千) flushes the pending coefficient (十 alone = 10). Returns ok=false on an unparseable rune
// or a non-positive result (conservative — an unparseable header simply is not stripped/rendered).
func parseSectionNumeral(s string) (int, bool) {
	section, num := 0, 0
	any := false
	for _, r := range s {
		switch {
		case r >= '0' && r <= '9':
			num = num*10 + int(r-'0')
			any = true
		case r >= '０' && r <= '９':
			num = num*10 + int(r-'０')
			any = true
		case lang.DefaultCJKSection().Zero[r]:
			num = num * 10
			any = true
		default:
			if d, isDigit := cjkSectionDigit(r); isDigit {
				num = num*10 + d
				any = true
				continue
			}
			if u, isUnit := cjkSectionUnit(r); isUnit {
				if num == 0 {
					num = 1
				}
				section += num * u
				num = 0
				any = true
				continue
			}
			return 0, false
		}
	}
	v := section + num
	if !any || v <= 0 {
		return 0, false
	}
	return v, true
}

// cjkSectionDigit / cjkSectionUnit read the shared lang.CJKSection (pair-14 §4): the SAME digit/unit value
// tables the ingest chapter-numeral inventory derives from, so there is ONE source of truth.
func cjkSectionDigit(r rune) (int, bool) {
	v, ok := lang.DefaultCJKSection().Digit[r]
	return v, ok
}

func cjkSectionUnit(r rune) (int, bool) {
	v, ok := lang.DefaultCJKSection().Unit[r]
	return v, ok
}

// chapterDraftChunks packs one chapter's paragraphs into DRAFT chunks (the fine tiling). Rule:
// pack whole paragraphs while the running OUTPUT-token estimate stays within DraftBudgetOut
// (prefer paragraph boundaries); a paragraph that alone exceeds the budget is flushed on its own
// and then SENTENCE-packed (never splitting a sentence; a single sentence over budget is its own
// OversizedSentence chunk). The running budget is tracked as ADDITIVE char-class counts (CJK vs
// other), applying est_out ONCE to the whole chunk — the "\n\n" separators are whitespace, which
// TokenClassCounts drops, so counting classes and estimating once reproduces est_out(joined text)
// exactly. Each emitted chunk gets its ChunkIdx (per-chapter) + EstOut; EditUnitID is set later.
func chapterDraftChunks(chapterNo int, paras []string, seg SegBudget, abbrevs map[string]bool) []Chunk {
	var out []Chunk
	chunkIdx := 0
	emit := func(text string, oversized bool, cjk, other int) {
		if t := strings.TrimSpace(text); t != "" {
			out = append(out, Chunk{
				Chapter: chapterNo, ChunkIdx: chunkIdx, Text: t,
				EstOut: seg.EstOut(cjk, other), OversizedSentence: oversized,
			})
			chunkIdx++
		}
	}
	var buf []string // paragraphs accumulated for the current chunk
	var bufCJK, bufOther int
	flush := func() {
		if len(buf) == 0 {
			return
		}
		emit(strings.Join(buf, "\n\n"), false, bufCJK, bufOther)
		buf = buf[:0]
		bufCJK, bufOther = 0, 0
	}
	for _, p := range paras {
		pCJK, pOther := TokenClassCounts(p)
		if seg.EstOut(pCJK, pOther) > seg.DraftBudgetOut {
			// Oversize paragraph: close any open chunk at this paragraph boundary,
			// then split the paragraph at sentence boundaries into its own chunks.
			flush()
			for _, sub := range packSentences(p, seg, abbrevs) {
				emit(sub.text, sub.oversizedSentence, sub.cjk, sub.other)
			}
			continue
		}
		// Normal paragraph: start a new chunk if adding it would exceed the budget
		// and the current chunk is non-empty (never split a paragraph here).
		if len(buf) > 0 && seg.EstOut(bufCJK+pCJK, bufOther+pOther) > seg.DraftBudgetOut {
			flush()
		}
		buf = append(buf, p)
		bufCJK, bufOther = bufCJK+pCJK, bufOther+pOther
	}
	flush()
	return out
}

// assignEditUnits groups a chapter's already-built DRAFT chunks into EDIT units — greedy packing of
// WHOLE draft chunks until the accumulated est_out would exceed EditCeilingOut — and stamps each
// chunk's EditUnitID (monotone across the book via *nextID). Because units are formed from whole
// draft chunks, a unit's source/draft span is exactly the concatenation of its member chunks (no
// straddle) and the edit wave reconstructs the unit's draft losslessly. A single draft chunk larger than the
// ceiling (an OversizedSentence, or a chapter whose first chunk already exceeds the ceiling) is its
// own unit — grouping never splits a chunk (never-split-below-chunk mirrors never-split-paragraph).
func assignEditUnits(chunks []Chunk, seg SegBudget, nextID *int) {
	acc := 0.0
	started := false
	for i := range chunks {
		if started && acc+chunks[i].EstOut > seg.EditCeilingOut {
			*nextID++ // close the current unit at this chunk boundary
			acc = 0
		}
		chunks[i].EditUnitID = *nextID
		acc += chunks[i].EstOut
		started = true
	}
	if started {
		*nextID++ // advance past the last unit so the next chapter starts a fresh id
	}
}

// subChunk is one sentence-packed sub-chunk with its char-class counts and the oversized-sentence
// flag (a single sentence whose est_out alone exceeds DraftBudgetOut).
type subChunk struct {
	text              string
	cjk, other        int
	oversizedSentence bool
}

// packSentences splits one oversize paragraph into sentence segments (lossless tiling —
// concatenation reproduces the paragraph) and packs consecutive segments into ≤ DraftBudgetOut
// (est_out) sub-chunks, never splitting a sentence. A single sentence larger than the budget
// becomes its own sub-chunk flagged OversizedSentence (passthrough, never clause-split — the plan's
// §2б decision: "degradation is not silent" is satisfied by the FLAG, not emergency clause machinery).
func packSentences(paragraph string, seg SegBudget, abbrevs map[string]bool) []subChunk {
	segs := splitSourceSentences(paragraph, abbrevs)
	var chunks []subChunk
	var buf strings.Builder
	var bufCJK, bufOther int
	flush := func() {
		if buf.Len() == 0 {
			return
		}
		chunks = append(chunks, subChunk{text: buf.String(), cjk: bufCJK, other: bufOther})
		buf.Reset()
		bufCJK, bufOther = 0, 0
	}
	for _, s := range segs {
		sCJK, sOther := TokenClassCounts(s)
		// A lone sentence over budget: flush what precedes it, then emit it as its own
		// OversizedSentence sub-chunk (never split a sentence, and never merge it with a
		// neighbour so the oversized flag stays attributable to the single sentence).
		if seg.EstOut(sCJK, sOther) > seg.DraftBudgetOut {
			flush()
			chunks = append(chunks, subChunk{text: s, cjk: sCJK, other: sOther, oversizedSentence: true})
			continue
		}
		if buf.Len() > 0 && seg.EstOut(bufCJK+sCJK, bufOther+sOther) > seg.DraftBudgetOut {
			flush()
		}
		buf.WriteString(s)
		bufCJK, bufOther = bufCJK+sCJK, bufOther+sOther
	}
	flush()
	return chunks
}

// TokenClassCounts counts the two token-estimate character classes of EstimateTokens
// (render.go): CJK ideographs/kana/hangul, and "other" non-space runes (whitespace
// folds into neighbours). Kept separate from EstimateTokens so the packer can sum
// counts ADDITIVELY across units and floor once at the chunk level. Mirrors
// EstimateTokens' classes; a recalibration there that changes packing granularity
// is itself a chunkerVersion bump (both are behaviour changes covered by the snapshot).
func TokenClassCounts(s string) (cjk, other int) {
	return text.DenseSparseCounts(s) // one shared sizing taxonomy (text.DenseScript)
}

// splitParagraphs breaks a chapter into paragraphs on blank lines and trims each,
// dropping empties. Deterministic; whitespace-only paragraphs vanish. (Ingest emits
// a blank line between block-level epub elements, so paragraphs survive extraction.)
func splitParagraphs(chapter string) []string {
	var paras []string
	for _, raw := range strings.Split(chapter, "\n\n") {
		if p := strings.TrimSpace(raw); p != "" {
			paras = append(paras, p)
		}
	}
	return paras
}

// --- source sentence segmentation (zh/ja/en) — chunkerVersion-versioned ---------

// splitSourceSentences segments one paragraph at sentence boundaries, returning
// substrings that TILE the input (their concatenation is exactly the paragraph —
// no character is lost or moved), so a packed sub-chunk is an exact source slice.
// A boundary is a run of sentence terminators, plus any closing quotes/brackets, at:
//
//	(1) a CJK terminator (。！？) — zero-width, CJK prose has no inter-sentence space —
//	    but ONLY at quote-depth 0. A terminator inside a quote is dialogue-internal:
//	    「止まれ！」と言った。 is ONE sentence ending at the outer 。, not two (the same
//	    for zh “…！”）— this is the dialogue case §3.7/D12-Q1 defers for the coverage
//	    oracle, but our source splitter is free to get it right;
//	(2) an ASCII terminator (.!?) or ellipsis (…) — only when followed by whitespace
//	    or end-of-paragraph (so a decimal/URL dot mid-token does not split, and en
//	    dialogue "Stop!" She ran. still splits on the space after the quote).
//
// Trailing whitespace after a boundary is attached to the LEFT segment (the next
// segment starts at the next content char), keeping the tiling exact. An
// abbreviation guard suppresses a split after a lone "." following a known
// abbreviation or a single-letter initial (never split "Mr. Smith" / "J. R. R.").
// When unsure it prefers NOT to split (over-merge is safe — units just pack
// together; over-split cuts a sentence, the forbidden case). Unbalanced quotes are
// tolerated (depth floors at 0), erring toward merge.
func splitSourceSentences(paragraph string, abbrevs map[string]bool) []string {
	runes := []rune(paragraph)
	var segs []string
	start, i, depth := 0, 0, 0
	for i < len(runes) {
		r := runes[i]
		if isOpenQuote(r) {
			depth++
			i++
			continue
		}
		if isCloseQuote(r) {
			if depth > 0 {
				depth--
			}
			i++
			continue
		}
		if !isSourceTerminator(r) {
			i++
			continue
		}
		// Consume a run of terminators (collapse "。。。" / "!?" into one boundary).
		j := i
		cjk := false
		for j < len(runes) && isSourceTerminator(runes[j]) {
			if isCJKTerminator(runes[j]) {
				cjk = true
			}
			j++
		}
		// The terminator's quote-depth is the depth HERE — before absorbing the
		// closing quote that may follow it (「…！」: the ！ is at depth 1, the 」 that
		// closes the quote comes after and must not retro-promote it to depth 0).
		depthHere := depth
		// Absorb a run of closing quotes/brackets that belong to the sentence,
		// decrementing quote-depth for the depth-tracked closers among them.
		e := j
		for e < len(runes) && isClosingBracket(runes[e]) {
			if isCloseQuote(runes[e]) && depth > 0 {
				depth--
			}
			e++
		}
		boundary := false
		if cjk {
			boundary = depthHere == 0 // dialogue-internal terminators do not end a sentence
		} else {
			// ASCII terminator / ellipsis: a boundary only outside a tracked quote AND
			// before whitespace or EOS. The quote-depth guard must apply here too, not
			// just to the CJK branch — otherwise 「Stop. Now.」 over-splits mid-dialogue
			// (external-review #4). Straight-quote en ("Stop!" She ran.) is untracked
			// (depth 0), so it still splits on the space as before.
			boundary = depthHere == 0 && (e >= len(runes) || isASCIISpace(runes[e]))
			// Abbreviation guard: a lone "." after an abbreviation/initial is not a end.
			if boundary && j-i == 1 && runes[i] == '.' && isAbbrevBefore(runes[start:i], abbrevs) {
				boundary = false
			}
		}
		if !boundary {
			i = e
			continue
		}
		// Attach the following whitespace run to the left segment (keeps tiling exact).
		w := e
		for w < len(runes) && isASCIISpace(runes[w]) {
			w++
		}
		segs = append(segs, string(runes[start:w]))
		start, i = w, w
	}
	if start < len(runes) {
		segs = append(segs, string(runes[start:]))
	}
	return segs
}

// isOpenQuote / isCloseQuote are the DEPTH-TRACKED paired quote/bracket delimiters
// (CJK brackets + curly quotes with distinct open/close glyphs). Straight quotes
// (" ') are NOT tracked — same glyph opens and closes, so depth is undecidable; en
// relies on the whitespace rule instead, which is correct for spaced prose.
func isOpenQuote(r rune) bool {
	switch r {
	case '「', '『', '（', '【', '《', '〈', '［', '｛', '“', '‘':
		return true
	}
	return false
}

func isCloseQuote(r rune) bool {
	switch r {
	case '」', '』', '）', '】', '》', '〉', '］', '｝', '”', '’':
		return true
	}
	return false
}

// isSourceTerminator is the sentence-terminator class for the SOURCE: CJK 。！？, ASCII .!?, and the
// ellipsis … (U+2026). The classes are DATA (lang.DefaultTerminators) shared with the coverage gate,
// which counts sentences over the same alphabet — the two used to carry a copy each.
func isSourceTerminator(r rune) bool { return lang.DefaultTerminators().IsTerminator(r) }

// isCJKTerminator is the zero-width-splitting subset (fullwidth CJK). The ellipsis
// is deliberately NOT here: "……" mid-CJK-sentence must not split, but "……。" still
// splits on the 。 in the same run.
func isCJKTerminator(r rune) bool { return lang.DefaultTerminators().IsCJK(r) }

// isClosingBracket is the closing-quote/bracket class absorbed into the left
// sentence after its terminator (ja/zh brackets + straight/curly quotes).
func isClosingBracket(r rune) bool {
	switch r {
	case '」', '』', '】', '）', ')', '》', '〉', '］', ']', '｝', '}',
		'"', '\'', '”', '’':
		return true
	}
	return false
}

// isASCIISpace is the whitespace used for ASCII-terminator boundary detection and
// tiling — the ordinary prose separators (space, tab, newline, CR, form feed,
// vertical tab, and the ideographic space U+3000).
func isASCIISpace(r rune) bool {
	switch r {
	case ' ', '\t', '\n', '\r', '\f', '\v', '　':
		return true
	}
	return false
}

// The abbreviation SET (trailing tokens after which a lone "." is an abbreviation, not a sentence end) is
// SOURCE data (internal/lang, embedded, sectioned per SOURCE language, pair-14 §4). It is now
// resolved from the BOOK's source language (lang.SentenceAbbrev(sourceLang), threaded through SplitChunks) —
// no longer the hard-coded "en" section — so a source whose ASCII period needs the guard supplies its own
// list, no Go edit. A CJK source (。 terminators) ships none, so the set is empty and never consulted. Single-
// letter initials are handled separately, script-agnostic, below.

// isAbbrevBefore reports whether the text immediately before a lone "." ends in a known abbreviation (from
// the source-language set) or a single-letter initial (so the "." is not a boundary).
func isAbbrevBefore(left []rune, abbrevs map[string]bool) bool {
	// Trailing run of ASCII letters (the "word" before the period).
	end := len(left)
	k := end
	for k > 0 && isASCIILetter(left[k-1]) {
		k--
	}
	word := left[k:end]
	if len(word) == 0 {
		return false
	}
	// Single-letter initial ("J.") — only when it stands alone (start or preceded by
	// a space/opening punctuation), never a one-letter word ending a real sentence
	// mid-flow would be rare; over-merge is the safe side.
	if len(word) == 1 {
		return k == 0 || !isASCIILetter(left[k-1])
	}
	return abbrevs[strings.ToLower(string(word))]
}

func isASCIILetter(r rune) bool {
	return (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z')
}