heaven/textmachine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
textmachine/docs/research/19-chunking-cohesion-sources.md

167 KiB
Raw Blame History

research/19 — приложение: источники и вердикты адверсариальной верификации

Сгенерировано сессией-ресёрчером 2026-07-16. Веб-фактура: 65-агентный workflow (8 тем + верификация + critic + добор). Формат: [LB]=несущий клейм, [M]=есть измерение. VERDICT — вердикт независимого верификатора по первоисточнику (CONFIRMED / OVERSTATED с коррекцией / UNVERIFIABLE); клеймы без VERDICT — не попали под кап верификации (5/тему).

t1-chunk-eng — Engineering best practices for splitting long documents for LLM processing — measured limitations of

  • [LB/M] In a controlled comparison of fixed-size vs breakpoint-based and clustering-based semantic chunking across document retrieval (10 datasets), evidence retrieval (5 RAGBench datasets), and answer generation, semantic chunking showed no consistent advantage: fixed-size won on most natural (non-stitched) datasets, semantic won only on artificially stitched high-topic-diversity corpora, and answer-generation differences were ~±0.01 BERTScore.
    • Источник (paper): https://arxiv.org/abs/2410.13070 — 'Is Semantic Chunking Worth the Computational Cost?' (arXiv 2410.13070, 2024)
    • Каветы: Entirely retrieval-pipeline-oriented (F1@5, retrieval-based QA with gpt-4o-mini on top-5 chunks); it does not test sequential full-coverage processing. The stitched-dataset wins for semantic chunking (e.g. Miracl F1@5 81.89% vs 69.45%) are on synthetic multi-topic concatenations unlike a single novel's chapter.
    • VERDICT: CONFIRMED
  • [ /M] The same study found embedding-model choice dominated chunking-strategy choice: the dunzhang/stella embedder gave a 7.44% average improvement (p=1.59e-5), larger than any gap between chunkers, and retrieval performance correlated only weakly with answer-generation quality.
    • Источник (paper): https://arxiv.org/html/2410.13070v1 — 'Is Semantic Chunking Worth the Computational Cost?' full text
    • Каветы: Same single origin as the previous finding — counts as one voice. The weak retrieval→generation coupling is itself a warning against transferring any retrieval-metric result to generation quality.
  • [ /M] Chroma's technical report (472 synthetic queries, 5 corpora, 328K tokens, token-level recall/precision/IoU): chunking strategy choice moves recall by up to ~9 points; a well-parametrized RecursiveCharacterTextSplitter (200 tokens, no overlap) reached 88.1% recall vs 91.9% for an LLM-based semantic chunker; reducing chunk overlap improved IoU while keeping comparable recall; the popular default of 800 tokens with 400-token overlap scored below average on recall and worst on all efficiency metrics.
    • Источник (practitioner): https://www.trychroma.com/research/evaluating-chunking — 'Evaluating Chunking Strategies for Retrieval', Chroma Technical Report, July 2024
    • Каветы: Vendor technical report (embedding-DB company), retrieval metrics only; recall/IoU have no analogue in full-coverage translation. Its overlap finding (overlap ≈ redundancy) is about embedding-index redundancy, not about generation context. Many 2025-26 chunking blog posts recycle these exact numbers — they are one voice, not independent confirmation.
  • [LB/M] LumberChunker, an LLM-driven segmenter built specifically for narrative books (iteratively prompting an LLM to find where content shifts), beat the strongest baseline by 7.37% DCG@20 on GutenQA (3,000 needle-in-haystack QA pairs over 100 Project Gutenberg narrative books).
    • Источник (paper): https://arxiv.org/abs/2406.17526 — 'LumberChunker: Long-Form Narrative Document Segmentation' (2024)
    • Каветы: This is the strongest published counter-evidence to 'deterministic chunking suffices for narrative' — but the gain is (a) retrieval-only (dense retrieval DCG, no generation task), (b) modest, and (c) bought with per-document LLM calls, which the design constraint (deterministic Go code, no model calls) forbids. No evidence it would improve sequential translation.
    • VERDICT: CONFIRMED
  • [LB/M] For literary translation specifically, translating whole paragraphs beats sentence-by-sentence translation: across 18 language pairs (incl. Japanese, Polish), with ~350 hours of professional-translator span-level error annotation, GPT-3.5 paragraph-level translation produced fewer mistranslations, grammar errors, and stylistic inconsistencies — but occasionally introduced content omissions absent in sentence-level mode.
    • Источник (paper): https://aclanthology.org/2023.wmt-1.41/ — Karpinska & Iyyer, 'Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist', WMT 2023
    • Каветы: GPT-3.5-era models; magnitude of reductions is in the paper's tables, not the abstract. The omission side-effect of longer translation units is directly relevant: larger chunks trade boundary errors for silent drops, which motivates an omission/coverage guard rather than maximal chunk size.
    • VERDICT: OVERSTATED — коррекция: For literary translation, translating whole paragraphs beats sentence-by-sentence translation: across 18 linguistically-diverse language pairs (incl. Japanese, Polish, English), in a human evaluation taking ~350 hours of annotation and analysis by hired bilingual translators providing span-level error annotations, GPT-3.5 (text-davinci-003) paragraph-level translation produced fewer mistranslations, grammar errors, and stylistic inconsistencies — but content omissions, while present in all systems, were more prominent in paragraph-level mode than in sentence-level mode.
  • [LB/M] A human study on En→Ru subtitles found 8.5% of translation pairs judged good in isolation (140 of 1,649) were wrong when read in context; of the identified context-dependent inconsistencies, deixis accounted for 37%, ellipsis 29%, and lexical cohesion 14% (~80% together; anaphora 6%, ambiguity 9%).
    • Источник (paper): https://aclanthology.org/P19-1116/ (arXiv:1905.05979) — Voita, Sennrich, Titov, 'When a Good Translation is Wrong in Context', ACL 2019
    • Каветы: 2019, pre-LLM sentence-level NMT, En→Ru OpenSubtitles — absolute rates for modern LLMs given in-prompt context are surely lower and unmeasured; but the taxonomy of WHAT breaks when a translation unit is cut off from prior context (pronouns/deixis, elided material, lexical/term cohesion) is the best measured inventory of chunk-boundary failure modes for translation, and Ru is the target language of this pipeline.
    • VERDICT: OVERSTATED — коррекция: A human study on 2,000 consecutive-sentence pairs of En→Ru subtitle translations found that in 7% of cases (140 of 2,000; equivalently ~7.8% of the 1,789 pairs whose sentences were individually good) the translations were good in isolation but bad in the context of each other; among these context-dependent inconsistencies, deixis accounted for 37%, ellipsis 29%, and lexical cohesion 14% (~80% together; ambiguity 9%, anaphora 6%, other 5%).
  • [LB/M] Translation quality degrades measurably with input length even well inside the context window: NLLB loses ~10 BLEU / ~0.2 COMET translating 256-token blocks vs isolated sentences; repetition-loop rate rises from 4% at 256 tokens to 81% at 2,048 tokens; TowerBase-7B (decoder, RoPE) degrades sharply past its 1,024-token training length; and within a document, sentences occurring later are translated systematically worse (position bias), with outputs also becoming too short.
    • Источник (paper): https://arxiv.org/html/2412.17592v2 — 'Investigating Length Issues in Document-level Machine Translation' (2024/25)
    • Каветы: Measured on NLLB and TowerBase-7B, i.e. small/specialized MT models, not frontier chat LLMs — the exact thresholds do not transfer, but the qualitative pattern (bigger chunk != free; repetition, truncation, and late-position degradation grow with chunk length) is corroborated by practitioner experience with long generations.
    • VERDICT: OVERSTATED — коррекция: Translation quality degrades measurably with input length even well inside the context window (EN->FR, TED pseudo-documents): NLLB-600M loses ~10 ds-BLEU / ~0.2 COMET translating 256-token blocks vs isolated sentences; the share of NLLB outputs containing long (>=10-gram) repetitions rises from 4% at 256 tokens to 81% at 2,048 tokens; TowerBase-7B (decoder, RoPE) shows a sharp BLEU decline past 1,024 tokens — well below its 4,096-token Llama2 context — with COMET already dropping at 256 tokens, which the authors attribute to continued pretraining mostly on isolated sentences; within a document, later sentences are translated systematically worse (position bias), and NLLB's outputs also become too short.
  • [ /M] In the closest published analogue to sequential full-coverage processing — book-length summarization by chunked traversal — 1,193 human annotations of GPT-4 summaries of 100 books yielded eight recurring coherence error types; incremental (running-state) processing benefited from larger chunks (Claude 2 at 88K-token chunks: 90.9 incremental vs 90.3 hierarchical BooookScore), while hierarchical merging did not.
    • Источник (paper): https://arxiv.org/abs/2310.00785 — Chang et al., 'BooookScore: A systematic exploration of book-length summarization in the era of LLMs', ICLR 2024
    • Каветы: Summarization compresses (errors of omission tolerated); translation must preserve everything, so 'bigger chunks help incremental workflows' collides with the doc-MT length-degradation evidence above. Treat as evidence that sequential chunked pipelines produce measurable cross-chunk coherence errors, not as a chunk-size prescription.
  • [LB/M] DelTA (ICLR 2025) shows that explicit deterministic state — a proper-noun record reusing each noun's first translation, plus bilingual summary and short/long-term memory — raises document-translation consistency scores by up to 4.58 points and COMET by up to 3.16 over strong long-context baselines across four LLMs and two doc-translation datasets, and keeps proper-noun translations consistent even across large distances in novels.
    • Источник (paper): https://arxiv.org/abs/2410.08143 — Wang et al., 'DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory', ICLR 2025
    • Каветы: DelTA translates sentence-by-sentence with memory (its own quality ceiling per Karpinska's paragraph findings); its memory components are LLM-maintained, not free. The transferable result: terminology cohesion across chunks is best solved by an explicit glossary/memory carried into every chunk's prompt, not by chunk overlap or bigger chunks.
  • [ /M] Industry practitioner baseline for RAG chunking is ~400-512 tokens with 10-20% overlap (Pinecone's oft-cited starting point: 512 tokens, 50-100 token overlap), with overlap justified as insurance against sentences split across chunk boundaries; recent systematic analyses find no universal optimum — optimal chunk size varies by dataset and document length (tested on NarrativeQA, NQ, NewsQA, SQuAD etc.), and wrong choices cost up to ~9% recall.
  • [ ] Sentence-window and parent-document patterns are officially documented as retrieval decoupling mechanisms: index/match on a small unit (single sentence; small child chunk) but hand the LLM a larger surrounding context at generation time (LlamaIndex SentenceWindowNodeParser stores a window of surrounding sentences in metadata and swaps it in before the LLM sees the node; parent-document retrieval returns the parent of a matched child).
  • [ /M] Follow-up empirical studies since 2410.13070 continue to find the semantic-vs-fixed trade-off unresolved and cost-dominated: a 2026 evaluation on academic texts (RAGAs framework) and a cost-focused study both report semantic chunking's coherence gains are 'not always proportional to the increased computational cost', with cluster-based semantic chunking reported roughly an order of magnitude slower to index than token-based splitting.
    • Источник (paper): https://arxiv.org/html/2606.00881v1 — 'Chunking Methods on Retrieval-Augmented Generation – Effectiveness Evaluation Against Computational Cost and Limitations'; https://arxiv.org/html/2607.01852v1 — 'Evaluating Chunking Strategies for RAG on Academic Texts'
    • Каветы: Skimmed via search snippets only, not deep-read; the widely repeated '~14x slower' figure could not be traced to a primary table and should be treated as unverified direction-of-effect, not a number. Both are again retrieval-pipeline studies.

t2-segmentation — Deterministic (non-LLM) text segmentation for narrative fiction: classic algorithms (TextTiling, C99

  • [ ] TextTiling (Hearst 1997) segments text into multi-paragraph subtopic units by sliding two adjacent bag-of-words blocks over the text, computing cosine lexical similarity, and placing boundaries at valleys (local minima) of the cohesion score; it is fully deterministic and was validated against human subtopic judgments on 12 expository texts.
  • [LB/M] C99 (Choi 2000) replaces TextTiling's raw similarity with a local rank matrix plus divisive clustering, and on Choi's synthetic benchmark (700 samples, each a concatenation of 10 random Brown-corpus segments) achieved 13% error vs 46% for TextTiling, 22% for DotPlot, 36% for Segmenter (3-11 sentence variant).
    • Источник (paper): https://arxiv.org/pdf/cs/0003083 — Choi, "Advances in domain independent linear text segmentation", NAACL 2000 (also http://www.cs.columbia.edu/~julia/papers/choi00.pdf)
    • Каветы: CRITICAL caveat: the benchmark concatenates unrelated documents, so 'boundaries' are trivially strong topic shifts; numbers do NOT transfer to within-document or narrative boundaries. TextTiling's 44-48% error even here shows raw TextTiling is weak out of the box.
    • VERDICT: CONFIRMED
  • [ ] BayesSeg (Eisenstein & Barzilay, EMNLP 2008) models each segment's words as draws from a segment-specific multinomial language model plus probabilistic cue-phrase models, maximizing observation likelihood to get a lexically-cohesive segmentation; a 2024 survey judges its results "still competitive" among non-neural methods.
    • Источник (paper): https://aclanthology.org/D08-1035/ — Eisenstein & Barzilay, "Bayesian Unsupervised Topic Segmentation"; survey characterization from https://arxiv.org/abs/2411.16613 — "Recent Trends in Linear Text Segmentation: a Survey" (EMNLP Findings 2024)
    • Каветы: Evaluated on expository/spoken corpora (textbooks, meetings), not fiction. I could not extract the original Pk numbers from the PDF; the 'still competitive' judgment is the survey authors' opinion.
  • [LB/M] GraphSeg (Glavaš et al., *SEM 2016), an unsupervised word-embedding graph method, scores excellently on the synthetic Choi dataset (Pk 5.6-7.2) but on natural Wikipedia documents (WIKI-50) gets Pk 63.56 — WORSE than a random baseline (52.65), i.e., embedding-similarity methods tuned on synthetic benchmarks can fail completely on real single-document segmentation.
    • Источник (paper): https://ar5iv.labs.arxiv.org/html/1803.09337 — Koshorek et al., "Text Segmentation as a Supervised Learning Task", NAACL 2018 (GraphSeg original: https://aclanthology.org/S16-2016/)
    • Каветы: Numbers as reported by Koshorek et al. (a competing paper), not by GraphSeg's authors; Pk is lower-is-better. Strongly warns against selecting a chunking algorithm based on Choi-dataset scores.
    • VERDICT: CONFIRMED
  • [LB/M] A supervised neural segmenter trained on 727,746 Wikipedia documents (WIKI-727K) achieves Pk 22.13 on the WIKI-727K test set and 18.24 on WIKI-50, while trained human annotators achieve Pk 14.97 on WIKI-50 — i.e., even massive supervision leaves ~20% window-error, and the human ceiling itself is far from 0.
    • Источник (paper): https://aclanthology.org/N18-2075/ and https://ar5iv.labs.arxiv.org/html/1803.09337 — Koshorek et al., "Text Segmentation as a Supervised Learning Task", NAACL 2018
    • Каветы: Wikipedia section boundaries, not narrative scenes; segmentation granularity is genuinely ambiguous even for humans, which bounds any algorithm's achievable 'accuracy'.
    • VERDICT: OVERSTATED — коррекция: A supervised neural segmenter trained on the 80% training split of the 727,746-document WIKI-727K corpus achieves Pk 22.13 on the WIKI-727K test set and 18.24 on WIKI-50, while human annotators (of unspecified training; the authors note they annotated only a few documents and 'lack familiarity with the right level of granularity', so 14.97 is a lower bound on the human gap, not a certified ceiling) achieve Pk 14.97 on WIKI-50 — massive supervision still leaves ~18-22% window-error, and even this human baseline is far from 0.
  • [LB/M] The first dedicated narrative scene segmentation corpus (Zehe et al., EACL 2021) defines a scene as a segment where story time ≈ discourse time and action, location, and character constellation stay constant; on 550k tokens of German dime novels (inter-annotator agreement gamma = 0.7) a BERT baseline reached only F1 24%.
    • Источник (paper): https://aclanthology.org/2021.eacl-main.276/ — Zehe et al., "Detecting Scenes in Fiction: A new Segmentation Task", EACL 2021
    • Каветы: German-language dime novels only; gamma=0.7 means humans themselves disagree substantially on scene boundaries. Same research group as STSS 2021 and the NAACL 2025 follow-up — treat the three as ONE research lineage, not independent confirmations.
    • VERDICT: CONFIRMED
  • [LB/M] In the STSS 2021 shared task (KONVENS), 5 teams trained on 20 annotated German dime novels; the best models reached F1 = 37% on in-domain dime novels (Track 1) and F1 = 26% on out-of-domain 19th-century highbrow literature (Track 2) — the organizers conclude scene segmentation is 'very challenging' but 'feasible in principle'.
    • Источник (paper): https://ceur-ws.org/Vol-3001/paper1.pdf and https://nilsreiter.de/publications/Zehe2021ab — Zehe et al., "Shared Task on Scene Segmentation @ KONVENS 2021"
    • Каветы: This is THE realistic accuracy anchor for narrative scene boundaries: ~1/3 F1 in-domain, ~1/4 out-of-domain. Same one-voice caveat (Würzburg/Zehe group). Exact metric tolerance definition not extracted from the PDF.
    • VERDICT: CONFIRMED
  • [ ] STSS 2021 participating systems used exactly the signal families relevant to a deterministic chunker: embedding-delta signals between adjacent windows, coreference chains of entities, temporal/spatial/entity feature vectors, twin-BERT embeddings + gradient-boosted trees, and sequential sentence classification — none exceeded 37% F1.
    • Источник (paper): https://ceur-ws.org/Vol-3001/ — CEUR-WS Vol-3001, STSS 2021 proceedings (papers by Schneider et al.; Hatzel & Biemann; Barth & Dönicke; Gombert; Kurfali & Wirén)
    • Каветы: Signal inventory inferred from paper titles and the volume index; per-team feature ablations not extracted.
  • [ ] The 2025 follow-up ("Assessing the State of the Art in Scene Segmentation", NAACL 2025) compared BERT-based and Llama-based models: Llama models are more robust across text types but slightly WORSE overall than BERT-based ones; a corrected training procedure gave a significant improvement, yet the task 'is very challenging for automatic methods, since it requires a high-level understanding of the text'.
    • Источник (paper): https://aclanthology.org/2025.naacl-long.500/ — "Assessing the State of the Art in Scene Segmentation", NAACL 2025
    • Каветы: Abstract-level claims only — exact F1 numbers could not be extracted (PDF did not parse). Same Zehe research lineage. Implication: even LLM-class models have not 'solved' scene segmentation as of 2025.
  • [LB/M] For Chinese, character/syllable n-gram variants of TextTiling not only work without word segmentation but BEAT word-based TextTiling: on the TDT2 Mandarin broadcast-news corpus, character-bigram TextTiling gave +8.84% relative F-measure and syllable bigrams +7.11% over word-based cohesion, and subword units are robust to word-segmentation ambiguity and OOV items.
  • [LB] The 2024 EMNLP-Findings survey of linear text segmentation states: unsupervised classics are generally outperformed by supervised transformer systems but remain relevant for low-resource settings; supervised models 'tend to overfit to specific cue phrases and domain-specific features'; and the standard metrics Pk and WindowDiff 'over-penalize near misses' and behave inconsistently in edge cases (Boundary Similarity recommended); literary text and non-English data are explicitly under-covered.
  • [ ] In English-language fiction manuscript convention, scene breaks are explicitly typographically marked: a blank-line break set off with '#' (or asterisks), with major vs minor breaks distinguishable — i.e., in edited fiction the scene boundary is usually a surface-visible token, not something to infer.
    • Источник (practitioner): https://www.shunn.net/format/scene_breaks/ — William Shunn, "Proper Manuscript Format: Scene Breaks"
    • Каветы: Publishing/manuscript convention, not a study; webnovel raw text (especially scraped/OCR zh sources) may lose or never contain these markers.
  • [LB] Japanese web novels (Syosetu/Kakuyomu culture) use blank lines pervasively as readability whitespace — guides recommend a blank line roughly every 5-10 lines and around dialogue — while ACTUAL scene changes are marked by dedicated separator symbols (＊, ◇, ◆, ■, ○, often repeated 1-3 times), a convention described as mainstream and low-cost for readers; vertical-phone reading makes dense text a psychological barrier.
  • [LB] Chinese webnovel chapters are short and mobile-optimized: ~2,000-5,000 characters per chapter (platform guide), ~3,000 characters per posted chapter with a chapter readable in ~10 minutes (Guangming Daily cultural-desk article), released 1-2+ chapters/day; total works run to millions of characters via '化整为零' (breaking the whole into small pieces).
  • [LB] Chinese webnovel prose convention has converged on very short paragraphs — one or two sentences per paragraph (一句一段) — explicitly attributed by Chinese writing guides to phone-screen reading fatigue; top authors are described as near-universally adopting one-sentence-per-paragraph, and paragraph rhythm is treated as a pacing device.
  • [ /M] Film/video scene boundary detection is likewise unsolved-but-improving on its own benchmark: MovieNet-SSeg annotates scene boundaries in 318 movies (train/val/test 190/64/64), evaluated by Average Precision; self-supervised ShotCoL raised SOTA AP by 13% relative (2021) and later multimodal methods (Scene-VLM, 2025) added +6 AP more — signals are shot-level visual/audio, not transferable to prose.

t3-docmt — Document-level MT and cross-sentence cohesion: measured shares of context-dependent errors, how much

  • [LB/M] In Voita et al. 2019 (en->ru OpenSubtitles), human annotation of 2000 consecutive-sentence pairs found 7% of pairs where each translation was good in isolation but wrong/inconsistent in context (a further 11% had at least one sentence-level error). Among those context-dependent inconsistencies: deixis 37%, ellipsis 29%, lexical cohesion 14% (the three together = 80%), ambiguity 9%, anaphora 6%, other 5%.
  • [LB/M] Voita et al.'s contrastive test sets place the disambiguating context within at most 3 previous sentences by construction: each instance = 3 context sentences + current sentence; deixis (3000 ex.) has 1000 examples each at antecedent distance 1/2/3 sentences back; lexical cohesion (2000 ex.) is distributed 855/630/515 across distances 1-3; CADec consumes up to 3 previous sentences on both source and target sides.
    • Источник (paper): https://github.com/lena-voita/good-translation-wrong-in-context (official data/code repo for ACL 2019 + EMNLP 2019 papers)
    • Каветы: The <=3-sentence distance is a test-set DESIGN choice, not a measurement of the natural distance distribution of cohesion dependencies; long-range dependencies (character names, honorifics, running metaphors in novels) are outside its scope. Same research group (Voita/Sennrich/Titov) as the previous finding — one voice.
    • VERDICT: CONFIRMED
  • [ /M] CADec (context-aware two-pass decoder refining a context-agnostic translation with 3 previous sentences) raised contrastive accuracy over the baseline: deixis 50.0%->81.6%, ellipsis-inflection 53.0%->72.2%, ellipsis-VP 28.4%->80.0%, lexical cohesion 45.9%->58.1%, with no BLEU loss.
  • [LB/M] DocRepair (Voita et al., EMNLP 2019): a MONOLINGUAL target-side repair model operating on sliding groups of 4 consecutive sentences (no source access) fixed most cohesion errors — deixis 91.8%, lexical cohesion 80.6%, ellipsis-inflection 86.4%, ellipsis-VP 75.2% (vs baseline 50.0/45.9/53.0/28.4) — with +0.69 BLEU (33.91->34.60); where annotators had a preference, 73% preferred DocRepair output.
    • Источник (paper): https://aclanthology.org/D19-1081/ (Context-Aware Monolingual Repair for Neural Machine Translation, EMNLP 2019; numbers via ar5iv https://ar5iv.labs.arxiv.org/html/1909.01383)
    • Каветы: Same authors as CADec/ACL-2019 — the whole Voita line is ONE voice. Trained via round-trip translation on monolingual data; en->ru subtitles. VP-ellipsis is the one phenomenon where source-less repair underperformed CADec (75.2 vs 80.0) — some repairs genuinely need the source.
    • VERDICT: OVERSTATED — коррекция: DocRepair (Voita et al., EMNLP-IJCNLP 2019): a MONOLINGUAL target-side repair model operating on groups of 4 consecutive sentences (no source access; the paper does not specify sliding/overlapping windows) scored deixis 91.8%, lexical cohesion 80.6%, ellipsis-inflection 86.4%, ellipsis-VP 75.2% on consistency test sets (vs baseline 50.0/45.9/53.0/28.4), with +0.69 BLEU (33.91->34.60) on 4-sentence fragments; among human-eval cases where annotators had a preference, 73% preferred DocRepair output.
  • [LB/M] Fernandes et al. 2021 (CXMI metric, en->de/en->fr IWSLT): the largest jump in measured context usage comes from adding the FIRST context sentence; additional sentences give diminishing returns, and target-side context is used more than source-side context. Context-aware word dropout increased usage and gave +0.5-1.4 BLEU plus contrastive-set gains.
    • Источник (paper): https://aclanthology.org/2021.acl-long.505/ (Measuring and Increasing Context Usage in Context-Aware Machine Translation, ACL 2021; arXiv https://arxiv.org/abs/2105.03482)
    • Каветы: Small NMT models on TED talks; 'models fail to use more context' is a property of THOSE models, not proof extra context is useless for LLMs. Fernandes/Neubig group overlaps with the MuDA paper below — related voice.
    • VERDICT: OVERSTATED — коррекция: Fernandes et al. 2021 (CXMI, en->de/en->fr IWSLT2017): the largest jump in measured context usage comes from adding the first context sentence, with diminishing (sometimes negative, source-side) returns for more sentences, and target-side context is slightly more used than source-side. Context-aware word dropout increased measured context usage and gave roughly +0.5 to +0.9 BLEU for non-pretrained contextual models (near-zero or slightly negative BLEU change with pretraining), plus clear contrastive-set gains in pronoun resolution and lexical cohesion.
  • [ /M] Kim, Tran & Ney 2019: in a manual analysis of a document-level NMT system's improvements, only ~5% of improved cases were attributable to genuine context use (pronoun resolution or topic-appropriate lexical choice); the rest looked like regularization effects, and feeding LONGER context degraded quality unless the context was filtered to content words.
    • Источник (paper): https://aclanthology.org/D19-6503/ (When and Why is Document-level Context Useful in Neural Machine Translation?, DiscoMT 2019); numbers via summary https://jlibovicky.github.io/2019/10/31/MT-Weekly-Document-Context.html
    • Каветы: The ~5% figure was verified against a blog summary, not the primary PDF (ar5iv conversion failed); primary abstract confirms the qualitative conclusions ('most improvements not interpretable as utilizing context', 'very long context not helpful'). 2019 RNN/Transformer NMT — LLM-era results (Karpinska below) partially supersede.
  • [LB/M] Cataphora/lookahead: Wong, Maruf & Haffari 2020 measured that ~16% of pronouns in the subtitles domain are cataphoric, ~37% of those resolvable within the single FOLLOWING sentence; an NMT model given one NEXT sentence as context beat the context-agnostic baseline (e.g. en->de BLEU 31.87->32.53, APT 60.8->61.4) and performed comparably to — sometimes better than — the same model given one PREVIOUS sentence, with significant gains on a targeted cataphora suite.
    • Источник (paper): https://aclanthology.org/2020.acl-main.530/ (Contextual Neural Machine Translation Improves Translation of Cataphoric Pronouns, ACL 2020; arXiv https://arxiv.org/abs/2004.09894)
    • Каветы: en<->de/pt/ru subtitles, not CJK literary prose; gains are modest in absolute BLEU. Still the clearest direct evidence that lookahead (next-sentence) context measurably helps.
    • VERDICT: CONFIRMED
  • [LB/M] Chinese zero pronouns: 26% of pronouns are dropped in the dialogue domain vs 7% in newswire; ZP-specialized translation methods move general BLEU barely (-0.4 to +0.6 avg) while pronoun-targeted APT jumps (+1.1 to +30.1); with oracle gold zero pronouns inserted, BLEU rises +3.4 and APT +63.4 — i.e., a large measured headroom that generic metrics hide. The survey judges current methods 'still far from real-world use'.
    • Источник (paper): https://arxiv.org/abs/2305.10196 (A Survey on Zero Pronoun Translation, Wang et al. 2023)
    • Каветы: Survey aggregates numbers largely from the same Tencent/Wang Longyue research line (also behind GuoFeng/WMT literary tasks) — partially one voice. Dialogue-heavy webnovels sit near the 26% end. For Japanese, an LREC 2020 dataset (https://aclanthology.org/2020.lrec-1.447/) shows context-aware NMT shortcomings on ja->en zero pronouns but publishes no headline error rate in the abstract.
  • [LB/M] LLM literary translation: Karpinska & Iyyer 2023 (GPT-3.5, 18 language pairs incl. ja/zh/ru, paragraphs of mean 7.45 sentences) — translating a WHOLE PARAGRAPH at once was preferred by professional translators over sentence-by-sentence in 71.1% of cases (82.8% vs Google Translate); sentence-level output had 29.5% more mistranslations, 65.4% more grammar errors, and over 12x more consistency errors (25 vs 2), yet critical errors persist even in paragraph mode.
    • Источник (paper): https://aclanthology.org/2023.wmt-1.41/ (Large language models effectively leverage document-level context for literary translation, but critical errors persist, WMT 2023; arXiv https://arxiv.org/abs/2304.03245)
    • Каветы: Paragraph-level (≈7 sentences), not chapter-level: the study does NOT show that even bigger single-call windows keep helping; text-davinci-003-era model.
  • [ /M] Par3 (Thai et al., EMNLP 2022; paragraph-aligned world literature): expert literary translators preferred human reference translations over Google Translate output 84% of the time; a GPT-3-based automatic post-editor's output was preferred over the raw MT 69% of the time — i.e., a post-editing/repair stage recovers a large share of literary quality deficit.
  • [LB/M] WMT 2023 Discourse-Level Literary Translation (GuoFeng webnovel corpus, zh->en: 1.9M sentences from 179 webnovels; test documents ~20 consecutive chapters, chapters averaging ~832-1.6K words): human MQM ranking and d-BLEU ranking DIVERGED — GPT-4 was 1st by human MQM (54.81) but only 3rd by d-BLEU (43.7), while the highest-d-BLEU system (52.2) was 2nd by human score; organizers conclude human evaluation is essential for literary discourse quality.
  • [ /M] Sun et al. 2022: naive single-run whole-document Doc2Doc concatenation translation 'yields disappointing translation and even fails' on some corpora (e.g., TED), while multi-resolution training (mixing document windows of several granularities with sentence data) makes Doc2Doc work and gives the best results — evidence that window/chunk granularity, not just total context, is the operative variable.
    • Источник (paper): https://aclanthology.org/2022.findings-acl.279/ (Rethinking Document-level Neural Machine Translation, Findings of ACL 2022)
    • Каветы: Encoder-decoder NMT trained from scratch; LLMs with long context behave differently, but 'Investigating Length Issues in Document-level MT' (https://arxiv.org/html/2412.17592) reports quality degradation with very long inputs for LLMs too.
  • [ /M] MuDA (Yin, Fernandes et al., ACL 2023) provides automatic taggers over 14 language pairs to find words whose translation REQUIRES context (pronouns, formality, lexical cohesion, verb form ...), confirming known phenomena and surfacing new ones; it is the standard tool if one wants to quantify, per corpus, how many tokens are context-dependent before choosing chunk boundaries.
    • Источник (paper): https://aclanthology.org/2023.acl-long.36/ (When Does Translation Require Context? A Data-driven, Multilingual Exploration; arXiv https://arxiv.org/abs/2109.07446)
    • Каветы: I could not extract the per-language frequency tables (fetch of full text failed), so exact percentages of context-dependent tokens per language remain unverified here; author overlap with Fernandes et al. 2021.
  • [ /M] Post & Junczys-Dowmunt 2023 argue and show that a standard Transformer with sufficient capacity handles document-level translation without bespoke architectures, with training data (document-level back-translation) and evaluation (generative contrastive metrics) being the real bottlenecks.
    • Источник (paper): https://arxiv.org/abs/2304.12959 (Escaping the sentence-level paradigm in machine translation)
    • Каветы: arXiv preprint (Microsoft Translator authors); partly a position paper — the architectural claim is measured, the paradigm claim is asserted.

t4-agentic-lit-mt — LLM multi-agent/pipeline systems for long literary translation: segmentation units, cross-segment ca

  • [ ] DelTA translates strictly sentence-by-sentence in an online (left-to-right) pass, explicitly to guarantee no sentence omissions and keep memory bounded, rather than translating whole chunks/documents per call.
    • Источник (paper): https://arxiv.org/abs/2410.08143 — DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory (ICLR 2025)
    • Каветы: Design description, not a result; sentence-level unit is the cost-maximal extreme of the segmentation spectrum.
  • [LB] DelTA carries four artifacts across segments: (1) Proper Noun Records — (source_noun, first_translation) pairs extracted by an LLM extractor and re-injected whenever the noun reappears, locking the FIRST translation; (2) a Bilingual Summary updated every m=20 sentences (source-side summary covers content/domain/style/tone, target-side covers content), maintained by summarize-then-merge LLM steps; (3) Long-Term Memory — the last l=20 source-target sentence pairs, from which an LLM retriever picks n=2 relevant pairs per query; (4) Short-Term Memory — the last k=3 sentence pairs verbatim.
    • Источник (paper): https://arxiv.org/html/2410.08143v1 — DelTA paper full text, memory-components section
    • Каветы: Exact m/l/k/n values extracted via automated read of the arXiv HTML; treat window constants as configuration choices of one paper, not established optima. Retrieval and summary-merge are themselves LLM calls (not deterministic).
    • VERDICT: CONFIRMED
  • [LB/M] DelTA's carried memory measurably improves cross-segment consistency and quality vs sentence-isolated and context-window baselines: proper-noun translation consistency (LTCR-1, exact-match agreement with the first translation of each noun) up by up to 4.58 percentage points and COMET up by up to 3.16 points on average, tested on four LLMs (GPT-3.5-Turbo-0125, GPT-4o-mini, Qwen2-7B-Instruct, Qwen2-72B-Instruct) on IWSLT2017 (8 language pairs) and GuoFeng Webnovel Zh→En.
    • Источник (paper): https://arxiv.org/abs/2410.08143 — DelTA (ICLR 2025)
    • Каветы: Self-reported by the one origin paper; 2024 models. One extracted table example (IWSLT2017, GPT-3.5, En→Zh: LTCR 77.06%→82.96%) exceeds the abstract's 'up to' figure — per-config numbers vary, so use the abstract-level range. LTCR-1 rewards consistency-with-first-translation, not correctness of that first translation.
    • VERDICT: CONFIRMED
  • [LB/M] DelTA's stated failure mode is inference cost/latency: auxiliary LLM components (extractor, retriever, summarizers) are 'frequently invoked' per sentence, giving prolonged runtime; its memory advantage over whole-document (Doc2Doc) prompting only materializes past roughly 70 sentences of document length.
    • Источник (paper): https://arxiv.org/html/2410.08143v1 — DelTA paper, limitations/efficiency discussion
    • Каветы: The 70-sentence crossover is about GPU memory for locally-served models, not API dollar cost; the runtime complaint is qualitative. Still, per-sentence agent loops multiply call count by sentences x auxiliary components.
    • VERDICT: CONFIRMED
  • [ ] TransAgents segments at CHAPTER level: the execution stage translates chapter-by-chapter over the WMT2023 discourse-level literary translation test set of 12 web novels x 20 consecutive chapters (240 chapters).
    • Источник (blog): https://gonzoml.substack.com/p/perhaps-beyond-human-translation — detailed review of (Perhaps) Beyond Human Translation (arXiv 2405.11804, TACL 2025)
    • Каветы: arXiv HTML and TACL page were unfetchable (LaTeXML shell / 403), so chapter-unit and test-set size are taken from a secondary review plus the WMT2023 task description; both trace to the one origin paper.
  • [LB] What TransAgents carries across chapters is a STATIC book-level 'translation guideline' built once in the preparation stage — glossary (Junior Editor over-collects candidate terms chapter-by-chapter, Senior Editor prunes common words, then translates terms in context), chapter summaries merged into a book/plot summary, and tone/style/target-audience decided from ONE randomly selected chapter; the execution stage (Translator → Localization Specialist → Proofreader, each paired with a Judgment critic agent) has no additional running inter-chapter memory, and the paper itself admits cross-chapter consistency 'remains a challenging task'.
    • Источник (blog): https://www.deeplearning.ai/the-batch/transagents-a-system-that-boosts-literary-translation-with-a-multi-agent-workflow — The Batch (deeplearning.ai) + https://gonzoml.substack.com/p/perhaps-beyond-human-translation
    • Каветы: Two secondary write-ups of ONE origin paper (arXiv 2405.11804) — one voice, not two. Key negative result: static book-level artifacts alone did not solve chapter-to-chapter consistency.
    • VERDICT: OVERSTATED — коррекция: What TransAgents carries across chapters is a STATIC book-level "translation guideline" built once in the preparation stage — glossary (Junior Editor as Addition agent exhaustively over-collects candidate terms per chapter, Senior Editor as Subtraction agent removes generic ones, then translates the terms in context), chapter summaries compiled into a book summary, and tone/style/target-audience set from one randomly selected chapter; in the execution stage (Translator → Localization Specialist → Proofreader as Action agents, each overseen by a Critique agent (Junior Editor) AND a Judgment agent (Senior Editor)) there is no additional running inter-chapter memory — the TACL version explicitly calls the static guidelines the system's "long-term memory," and the only other cross-chapter mechanism is a final review that checks flow between adjacent chapters. However, the paper does NOT admit that its cross-chapter consistency "remains a challenging task": it frames book-wide coherence/cohesion/consistency as "one of the principal challenges in literary translation" that its guidelines are designed to address, and its case study claims TransAgents maintains global consistency where GPT-4 does not.
  • [ /M] TransAgents' measured evaluation is preference-based and genre-unstable: monolingual human raters preferred it over human reference and GPT-4 only on average (77.8% preference in fantasy-romance vs 39.1% in science fiction); GPT-4-judge bilingual preference over the human Reference 1 ran roughly 66% vs 30%; meanwhile d-BLEU collapsed to 25.0±2.4 vs 47.8 for direct GPT-4 and 47.3 for Google Translate.
  • [LB] TransAgents' documented failure mode is content omission and free interpretation: professional translators inspecting outputs noted it 'sometimes losing parts of the original text' (including chapter-title omissions), and The Batch summarizes outputs as 'less-precise interpretations' rather than faithful translations — the multi-agent rewrite loop trades fidelity for fluency.
    • Источник (blog): https://gonzoml.substack.com/p/perhaps-beyond-human-translation + https://www.deeplearning.ai/the-batch/transagents-a-system-that-boosts-literary-translation-with-a-multi-agent-workflow
    • Каветы: Qualitative observations from the origin paper's own analysis, relayed by secondary sources; the d-BLEU gap (25.0 vs 47.8) is the quantitative shadow of the same phenomenon.
    • VERDICT: OVERSTATED — коррекция: TransAgents' documented failure mode is content omission and free interpretation: the paper finds both gpt-4-1106-preview and TransAgents "exhibit significant issues with content omission" in chapter segments (chapter titles, by contrast, were rendered consistently by TransAgents — the inconsistency fault there was GPT-4's), and the paper summarizes professional translators' comments as a novel-like expressive style that "sometimes omits parts of the original text"; The Batch describes the outputs as "not word-by-word translations of the inputs but less-precise interpretations" — the pipeline trades fidelity for fluency, though omission was equally present in the vanilla GPT-4 baseline.
  • [ /M] TransAgents' cost was about $2.08 per chapter vs $168.48 per chapter for professional human translation (the ~80x cost-reduction claim), using GPT-4-1106-preview as the backbone.
  • [ ] TransAgents' human evaluation is methodologically weak: raters were monolingual (never saw the source), judged isolated ~150-word segments drawn only from the first two chapters of each novel, and gave bare preferences with no error typology (unlike MQM) — so the 'preferred over human translation' headline cannot be read as fidelity or book-level cohesion evidence.
  • [ /M] Independent evidence that preference-style evaluation overstates LLM literary translation: on LITEVAL-CORPUS (2k+ paragraph-level translations, 4 language pairs, 9 systems incl. GPT-4o/DeepL), professional translators preferred human translations ~94.4% of the time over the best MT/LLM outputs (100% in De-En and De-Zh), and standard MQM was found inadequate for literary translation while evaluator expertise strongly controlled adequacy.
    • Источник (paper): https://arxiv.org/abs/2410.18697 — How Good Are LLMs for Literary Translation, Really? (LITEVAL-CORPUS, NAACL 2025)
    • Каветы: Does not include TransAgents/multi-agent systems; paragraph-level (mean 5.7 sentences), authors admit it does not capture document-level nuance. Directly undercuts monolingual-preference claims of human parity.
  • [ ] Andrew Ng's translation-agent chunks purely deterministically — RecursiveCharacterTextSplitter.from_tiktoken_encoder(model_name="gpt-4") with MAX_TOKENS_PER_CHUNK=1000 and an evening function (total_tokens / ceil(total/limit)), zero overlap — and carries NOTHING between chunks: every chunk's prompt embeds the FULL source text with the current chunk wrapped in <TRANSLATE_THIS> tags; no glossary, no summary, no previous translations cross chunks; the translate→reflect→improve loop is entirely within-chunk. No quantitative evaluation is published.
    • Источник (practitioner): https://github.com/andrewyng/translation-agent — translation-agent repo, src/translation_agent/utils.py
    • Каветы: Explicitly 'not mature software' (weekend demo). Full-source-per-chunk is O(N²) total prompt tokens over a book and offers no terminology locking — it is a non-solution for book-scale cohesion, but its splitter pattern is a precedent for deterministic Go chunking.
  • [LB/M] At WMT24's discourse-level literary translation task (GuoFeng Webnovel V2; Zh→En test = 12 books, ~239 chapters, ~28.1K words per document), the winning recipes were terminology control plus editor/critic agents on top of chunk translation — NLP2CT-UM: GPT-4o multi-candidate generation with 'a custom terminology table' + xCOMET selection (human General score 3.96/5); SJTU-LoveFiction: chunk-based SFT + a GPT-4o Translation Editor Agent + a 'Term Proofreader Agent' (Discourse 4.13/5) — and NO system reached human translator quality; automatic and human rankings diverged sharply (NTU: sentence BLEU 20.9 but 2.58/5 human).
    • Источник (paper): https://arxiv.org/html/2412.11732v1 — Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation (also https://aclanthology.org/2024.wmt-1.58/)
    • Каветы: Shared-task findings aggregate 10 submissions from 5 teams — multiple independent voices, strongest external validation available that glossary/terminology enforcement is the load-bearing cross-segment artifact for webnovels. Human eval by professional translators (Cohen's kappa 0.86).
  • [ /M] DITING (Oct 2025), the first webnovel-translation-specific benchmark (18K+ expert-annotated Zh-En pairs), finds the discriminating error classes are exactly cross-segment/cultural ones — idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, cultural safety; DeepSeek-V3 scored best overall (5.16/6, GPT-4o 5.09), Chinese-trained LLMs beat larger Western models (Qwen3-8B 3.96 > LLaMA3-70B 3.58); its own AgentEval judge correlates with humans at Spearman 0.669 vs ~0.47 for BLEU/COMET/BLEURT; the authors admit the benchmark is sentence-level and 'overlook[s] document-level narrative coherence'.
    • Источник (paper): https://arxiv.org/abs/2510.09116 — DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation
    • Каветы: Single paper, 2025 models; per-dimension scores are LLM/expert-rubric based, not production reading data. Its Chinese-model advantage is task-specific (Zh webnovel domain) — do not generalize to zh→ru.
  • [ ] 2026 successors are moving from fixed context windows to SELECTED context: G²C-MT frames choosing which previous paragraphs to feed the translator as path discovery over a lightweight discourse graph (nodes = paragraphs; edges = semantic similarity, adjacency, keyword overlap) and reports beating strong baselines on DeepSeek-V3, Gemini-2.5-Flash-lite and Qwen-2.5/3; SAMAS (Feb 2026) is a multi-agent system carrying an extracted style 'spectrum' across a literary work.
    • Источник (paper): https://arxiv.org/abs/2606.03078 — G²C-MT: Graph-Guided Context Selection for Document-Level Machine Translation; https://arxiv.org/pdf/2602.19840 — SAMAS
    • Каветы: G²C-MT numeric deltas and SAMAS results were not extractable from fetched excerpts — treat as direction-of-field evidence, not measured gains. Notably G²C-MT's edge features (adjacency, keyword/entity overlap) are computable deterministically without model calls.

t5-longcontext — Long-context and long-output LLM degradation evidence for chunk sizing and prompt layout in a book-t

  • [LB/M] Lost in the Middle (Liu et al., TACL 2024) measured a U-shaped position effect on multi-document QA and synthetic key-value retrieval ONLY, at modest input sizes (10/20/30 documents, ~3-5K tokens): GPT-3.5-Turbo with 20 documents scored 75.8% with the answer at the beginning vs 53.8% in the middle (a >20-point drop), falling BELOW its 56.1% closed-book baseline. It did not test open-ended generation or translation.
    • Источник (paper): https://arxiv.org/abs/2307.03172 — Lost in the Middle: How Language Models Use Long Contexts (Liu et al., TACL 2024); full text via https://ar5iv.labs.arxiv.org/html/2307.03172
    • Каветы: Extractive retrieval/QA tasks on 2023 models (GPT-3.5, Claude-1.3, MPT-30B, LongChat-13B); the paper says nothing about generation quality vs position. Widely over-generalized in secondary literature to all long-context use.
    • VERDICT: CONFIRMED
  • [ /M] The same paper measured a prompt-layout lever: query-aware contextualization (placing the query both BEFORE and AFTER the long context) took GPT-3.5-Turbo-16K key-value retrieval from a worst case of 45.6% to perfect on 300 pairs — but the paper found it barely changed multi-document QA.
    • Источник (paper): https://arxiv.org/abs/2307.03172 — Lost in the Middle (Liu et al., TACL 2024)
    • Каветы: Effect proven on synthetic key-value retrieval; minimal effect on QA; untested for translation. Treat instruction-repetition after the chunk as a cheap hypothesis to A/B, not established fact.
  • [LB/M] RULER (Hsieh et al., 2024) showed effective context length is far below advertised: of 17 models all claiming >=32K context, only about half maintained satisfactory performance at 32K on its 13 tasks (retrieval variants, multi-hop tracing, aggregation), despite near-perfect scores on vanilla needle-in-a-haystack.
    • Источник (paper): https://arxiv.org/abs/2404.06654 — RULER: What's the Real Context Size of Your Long-Context Language Models?
    • Каветы: 2024 open and closed models; synthetic tasks. Establishes the 'advertised != effective' principle, not translation-specific numbers.
    • VERDICT: CONFIRMED
  • [LB/M] NoLiMa (Adobe Research, 2025) showed degradation is much worse when the query and the relevant text share minimal lexical overlap (association must be inferred): of 13 models claiming >=128K context, 11 dropped below 50% of their short-context baseline at 32K; GPT-4o fell from 99.3% to 69.7%; GPT-4.1's effective length (>=85% of base score) was ~16K vs a claimed 1M; Gemini 2.0 Flash dropped to 16.4% at 128K.
    • Источник (paper): https://arxiv.org/abs/2502.05167 — NoLiMa: Long-Context Evaluation Beyond Literal Matching; effective-length table at https://github.com/adobe-research/NoLiMa
    • Каветы: Needle-retrieval task, not generation. Highly relevant anyway: cross-lingual glossary/memory lookups are inherently low-lexical-overlap matches, the exact regime NoLiMa shows collapses fastest.
    • VERDICT: CONFIRMED
  • [LB/M] Chroma's 'Context Rot' report (July 2025) tested 18 then-current models (Claude 4 Opus/Sonnet, Gemini 2.5 Pro/Flash, o3, GPT-4.1 family, Qwen3) and found all still degrade non-uniformly as input grows, well before documented limits; counterintuitively, models performed BETTER on shuffled haystacks than on logically coherent ones, and low needle-question similarity amplified length degradation.
    • Источник (practitioner): https://research.trychroma.com/context-rot — Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma, Hong/Troynikov/Huber)
    • Каветы: Company research report, not peer-reviewed; single origin voice despite many blogs citing it. Answers the 'do 2025 frontier models still degrade mid-context' question: yes, though less than 2023 models.
    • VERDICT: CONFIRMED
  • [LB/M] Chroma's Repeated Words task is direct long-OUTPUT evidence on 2025 frontier models: asked to replicate text where output length scales with input (up to 10,000 words), all 18 models degraded with length, placed a unique word correctly more reliably when it appeared early, and at long lengths frequently under-generated or hallucinated.
    • Источник (practitioner): https://research.trychroma.com/context-rot — Context Rot (Chroma, 2025)
    • Каветы: Verbatim replication is a proxy, not translation; but it is the closest published analog to 'output must be input-sized', which is the translation-chunk regime.
    • VERDICT: OVERSTATED — коррекция: Chroma's Repeated Words task (Context Rot, 2025) is direct long-output evidence: models replicate text whose output length is directly proportional to input length (inputs up to 10,000 words). Among the models evaluated on this task — a subset of the report's 18 LLMs, since o3 and GPT-3.5 turbo were excluded — performance consistently degraded as length increased; accuracy was highest when the unique word appeared near the beginning of the sequence (especially at longer inputs); and at long lengths models often under-generated (repeating words until hitting the output token limit) or produced random words not present in the input.
  • [LB/M] Fiction.LiveBench (narrative deep-comprehension over stories, the closest input-side benchmark to fiction translation) shows accuracy sliding well before the max window — typically from ~32K — with only a few models (Gemini 2.5 Pro, GPT-5, Grok 4 reported) holding good accuracy at 192K, while e.g. DeepSeek V3.1 and Claude Sonnet 4 (thinking) dropped by ~120K.
  • [ ] 2026 theory work argues the U-shaped position bias is architectural — present at initialization and not removed by standard training: causal masking concentrates gradient influence at the prompt start and residual connections anchor the final token, leaving a structurally disfavored middle — so mid-context weakness should be engineered around, not waited out.
    • Источник (paper): https://arxiv.org/abs/2603.10123 — Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias; also https://arxiv.org/html/2602.16837v2 — A Structural Theory of Position Bias in Transformers
    • Каветы: Theoretical analyses on small/untrained models (GPT-2, Qwen2 scale); corroborates but does not measure frontier-model behavior.
  • [LB/M] For translation specifically, quality degrades with document length and the degradation concentrates in LATER sentences: Pombal-adjacent work (Herold et al. style; arXiv 2412.17592) measured NLLB-600M losing ~10 d-BLEU and ~0.2 COMET translating 256-token documents vs isolated sentences, quality systematically lower for sentences deeper in the document, and degenerate 10-gram repetition loops rising from 4% of pseudo-documents at 256 tokens to 81% at 2048 tokens.
    • Источник (paper): https://arxiv.org/abs/2412.17592 — Investigating Length Issues in Document-level Machine Translation (full text https://arxiv.org/html/2412.17592v2)
    • Каветы: Models are NLLB-200-600M (encoder-decoder) and TowerBase-7B — far below the pipeline's model class; TED-talk domain, not literary zh->ru. Direction of effect (late-chunk degradation, repetition onset past training-length) is the transferable part, not the absolute thresholds.
  • [LB/M] SEGALE (arXiv 2509.17249, book-length MT evaluation) measured seven open-weight LLMs translating long documents: most exhibited sharp quality degradation at 4K or 8K context; EuroLLM-9B at 4K stopped translating and summarized instead; under-/over-translation (omission) errors played a significant role in the degradation; Qwen2.5-72B-Instruct was the exception, stable (slightly improving) up to 4K.
    • Источник (paper): https://arxiv.org/html/2509.17249v1 — Extending Automatic Machine Translation Evaluation to Book-Length Documents (SEGALE)
    • Каветы: Open-weight models up to 72B, not frontier API models (no DeepSeek V4/GLM-5/Gemini numbers); still the most direct published quality-vs-chunk-size measurement for translation. Larger/stronger models (Qwen-72B) visibly push the cliff outward.
  • [LB/M] LongWriter (arXiv 2408.07055) measured a hard output-length ceiling in 2024-era aligned models: models accepting 100K-token inputs struggled to generate beyond ~2,000 words, traced to scarcity of long outputs in SFT data, and fixable by adding long-output SFT examples (scaling coherent output past 10,000 words).
    • Источник (paper): https://arxiv.org/abs/2408.07055 — LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
    • Каветы: Measured on 2024 models (GLM-4, GPT-4o era) with open-ended writing tasks; 2026 frontier models have raised caps and long-output tuning, so the exact 2K-word ceiling should not be assumed — but the asymmetry (usable output << usable input) is corroborated by LongProc and Chroma on newer models.
  • [LB/M] LongProc (Princeton, arXiv 2501.05414) measured long structured GENERATION with rule-based scoring on 17 models at 500/2K/8K output tokens: open-weight models typically faltered already at 2K-token outputs, and even closed models like GPT-4o showed significant degradation at 8K-token outputs, despite all claiming >32K context windows.
  • [LB/M] Documented provider output caps for the pipeline's model class (retrieved July 2026) are far above the quality-degradation thresholds: DeepSeek v4-flash/pro — 1M context, max output 384K tokens (official docs; deepseek-chat/reasoner aliases deprecated 2026-07-24); Gemini 2.5 Pro — input limit 1,048,576, output limit 65,536 (official docs); GLM-5.2 — 1M context, up to 131,072 output tokens (vendor-adjacent listings).
  • [ ] The field's literary-translation evaluation standard (WMT24 Discourse-Level Literary Translation, GuoFeng webnovel corpus, zh->en/de/ru) treats one webnovel CHAPTER as the document unit, with official ranking by human judgment over ~20 consecutive chapters per book — i.e., chapter-level coherence, not sentence-level metrics, is the accepted quality frame for this exact domain.
    • Источник (paper): https://arxiv.org/html/2412.11732v1 — Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation
    • Каветы: Setup/context claim, not a degradation number; the zh->de and zh->ru references were produced by GPT-4 translation plus human post-editing, so zh->ru 'gold' data there is machine-flavored.

t6-cat-practice — How professional translation practice handles segmentation and context: SRX, translation units, CAT

  • [ ] SRX (Segmentation Rules eXchange) is an XML-based standard (OSCAR 1.0 in 2004, 2.0 in 2008; maintained by GALA after LISA's 2011 insolvency) that defines segmentation as ordered ICU-regex break/exception rules; it was created because tools segmented text differently, making TMX exchange unreliable.
    • Источник (standard): https://en.wikipedia.org/wiki/Segmentation_Rules_eXchange (Segmentation Rules eXchange — Wikipedia); see also https://www.gala-global.org/srx-10 (SRX 1.0 | GALA Global)
    • Каветы: Default rules break on full stop/colon/question/exclamation marks with exception patterns — i.e., sentence-level is the industry default unit. This validates that deterministic regex-driven segmentation without model calls is exactly how the industry itself does it.
  • [LB] The classic criticism of sentence-level TM/CAT segmentation is loss of text-level cohesion: Heyn's (1998) 'peephole' effect (translators avoid intratextual references so segments stay reusable) and the 'sentence-salad' effect (incoherent text assembled from segments), plus the observation that translators start working 'mechanically sentence-by-sentence, instead of focusing on how each sentence relates to those around it and to the text as a whole'; TM has therefore 'not been considered appropriate for literary or creative texts'.
  • [LB/M] In the only controlled study found directly on point (Vieira, Zelenka, Youdale, Zhang & Carl 2023, Translation & Interpreting 15(1)): English→Chinese post-editing of science-fiction short stories in Trados Studio compared sentence-based vs paragraph-based presentation; paragraph segmentation was associated with lower technical (editing) effort on average, MT assistance did not substantially shorten translation time, and inter-translator variation was large.
    • Источник (paper): https://www.trans-int.org/index.php/transint/article/view/1389 (Translating science fiction in a CAT tool: machine translation and segmentation settings — Vieira et al. 2023)
    • Каветы: Single study, small N (typical for PE process research), one language pair (EN→ZH — the reverse of your zh→ru direction), sci-fi short stories; exact effect sizes were not retrievable from the abstract page. Do not generalize beyond 'paragraph presentation is at least competitive and likely better for literary PE'.
    • VERDICT: CONFIRMED
  • [LB] Vinay & Darbelnet (1958) define the unit of translation as a 'unit of thought' — the smallest segment of the utterance whose signs are so linked that they must not be translated individually — explicitly rejecting the word; later scholarship (Barkhudarov: whole text as unit for poetry; Bassnett: text as prime unit for prose; Koller: the more dissimilar the languages, the larger the unit needed) pushes the literary unit toward paragraph/text level, and larger units are held to yield more idiomatic translation.
    • Источник (paper): https://en.wikipedia.org/wiki/Translation_unit (Translation unit — Wikipedia, citing Vinay & Darbelnet, Barkhudarov, Bassnett, Koller)
    • Каветы: Theoretical, not empirical. Koller's point is directly relevant to zh/ja→ru: distant language pairs need larger units, i.e., sentence-aligned chunking is the worst case precisely for CJK→European pairs.
    • VERDICT: OVERSTATED — коррекция: Vinay & Darbelnet (1958; English trans. 1995, p. 21) define the unit of translation — equated with the 'lexicological unit' and 'unit of thought' — as "the smallest segment of the utterance whose signs are linked in such a way that they should not be translated individually" (per Munday/Hatim & Munday; some renderings say "literally"), and per standard translation-studies accounts they reject the word as the unit of translation [source: V&D 1958/1995:21 via Hatim & Munday 2004:138 — NOT the Wikipedia article, which only mentions V&D discussing the single word via Saussure]. Separately, the Wikipedia 'Translation unit' article does support: Barkhudarov (translation unit can be a complete text, e.g. in poetry), Bassnett (text as prime unit for prose), Koller (the more unrelated the languages, the larger the unit), and that larger translation units give a better chance of idiomatic translation.
  • [LB] CAT tools operationalize 'context' minimally as the immediately adjacent segments: memoQ's official docs define a context match (101%) as a 100% match whose stored previous AND next source segments also match, and a 102% 'double context' match when both text-flow context and structural ID match; context is only checked within one structural unit (e.g., a cell/heading has empty prev/next context at its edges).
  • [ /M] Ruffo's survey of 150 literary translators from 35 countries found only ~25% use CAT tools for literary work; attitudes were positive toward generic tools and even TMs (49% positive on technology overall) but negative toward MT/AI, with respondents saying CAT/MT are unsuitable for literary translation because creativity and lexical/syntactic variation are compromised.
    • Источник (paper): https://benjamins.com/catalog/tris.23015.ruf (Ruffo, 'Literary translators in-between: self-imaging discourse and relationship to technology', Translation in Society); survey figures relayed via https://aclanthology.org/2023.eamt-1.48.pdf (DUAL-T, EAMT 2023)
    • Каветы: Self-report survey, not performance data; percentages retrieved via search-result summaries of the DUAL-T/Ruffo publications rather than the paywalled full text.
  • [LB] Netflix's Timed Text Style Guide is a production-grade example of deterministic sense-block segmentation: max 42 chars/line, max 2 lines, keep to one line unless over limit, break 'after punctuation marks, before conjunctions, before prepositions', and never separate article from noun, adjective from noun, first from last name, verb from subject pronoun, prepositional verb from its preposition, or verb from auxiliary/reflexive/negation; the general guide adds that subtitles should be segmented at clause level.
    • Источник (official-doc): https://partnerhelp.netflixstudios.com/hc/en-us/articles/217350977-English-USA-Timed-Text-Style-Guide (Netflix English (USA) Timed Text Style Guide); https://partnerhelp.netflixstudios.com/hc/en-us/articles/215758617-Timed-Text-Style-Guide-General-Requirements
    • Каветы: Subtitling is a different task (display constraint, not translation-unit choice), but it proves that 'break only at natural linguistic boundaries, expressed as a finite rule list' is an accepted, fully deterministic industry practice — a direct design precedent for rule-based chunk boundary scoring in Go.
    • VERDICT: OVERSTATED — коррекция: Netflix's Timed Text Style Guides prescribe rule-based line treatment: the English (USA) guide sets max 42 characters/line, max 2 lines, keep to one line unless over the limit, break after punctuation marks / before conjunctions / before prepositions, and states the line break should not separate an article from its noun, adjective from noun, first from last name, verb from subject pronoun, prepositional verb from its preposition, or verb from auxiliary/reflexive pronoun/negation. The clause-level rule comes from a separate document — the Subtitle Templates guide (Section 25), which requires that sentences split across multiple subtitle events "are always segmented at a clause level" — not from the General Requirements guide, and neither document uses the term "sense block".
  • [LB] CJK webnovel translation (fan and commercial) is organized around the CHAPTER as the work unit — serialized 1–2 chapters/day on Qidian/JJWXC-style platforms, translated and released chapter-by-chapter on platforms like Wuxiaworld (still the most influential fan-translation platform per Deng 2024) — with a persistent master glossary as the central consistency instrument shared by translator, editor, and proofreader.
  • [ ] Terminology 'consistency drift' across chapters (a name or cultivation term rendered differently by chapter 20/47) is described by practitioners as THE defining failure mode of amateur/machine webnovel translation, and tools in this niche compete on glossary managers and pre-analysis passes that lock names/terms before translating.
  • [ /M] DITING (Zhang et al., arXiv 2510.09116, 2025), the first evaluation framework specifically for web-novel MT, formalizes six webnovel-specific quality dimensions — idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, cultural safety — on 18K+ expert-annotated zh-en sentence pairs; among 14 models, Chinese-trained LLMs outperformed larger foreign models and DeepSeek-V3 was most faithful/stylistically coherent.
    • Источник (paper): https://arxiv.org/abs/2510.09116 (DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation)
    • Каветы: zh→EN, not zh→RU; benchmark is sentence-pair based, so it measures error types (zero pronouns, terminology) rather than chunking policy; single paper, not yet independently replicated. Its dimension list is still the best available checklist of what chunk-level context must preserve.
  • [LB] Chinese and English differ in paragraph organization and dialogue layout: in Chinese fiction dialogue is often embedded inside narrative paragraphs, while English convention (Chicago-style craft guidance) starts a new paragraph per speaker; translation-studies work on Chinese fiction states translators must consider target-language habits and split/merge paragraphs accordingly, moving dialogue into separate paragraphs for visual clarity.
  • [LB] Chinese webnovels are written mobile-first with very short, left-aligned paragraphs and heavy white space (serialized scrolling platforms), so their paragraph rhythm is a platform artifact, not an authorial prose unit — meaning source paragraph boundaries in webnovels are dense, shallow signals rather than European-style discourse paragraphs.
  • [LB] Japanese fiction formats each dialogue utterance in 「」 kagikakko, typically starting on its own line without indent, with terminal punctuation inside the bracket and frequently NO speaker attribution — readers infer speakers from register, sentence-final particles, and gendered speech; translators into European languages must decide whether to add attributions (Murakami is noted for atypically heavy tag use giving a 'foreign feel').
  • [ /M] Empirical follow-up exists for the cohesion criticism: a journal study specifically measured the impact of TM sentence segmentation on cohesion and coherence of EN→AR technical translations, explicitly building on the peephole/sentence-salad effects — i.e., the segmentation-cohesion link is treated as a testable degradation, not just an opinion.
    • Источник (paper): https://jwadi.journals.ekb.eg/article_268202.html (The Impact of Translation Memory Segmentation on the Cohesion and Coherence of Technical Texts Translated from English into Arabic)
    • Каветы: Article page failed to load (socket hang up); claim rests on the search-result snippet and title. Technical texts, not literary; EN→AR. Counts as corroboration of the Heyn line, but it is still downstream of the same single origin.

t7-caching — Official provider documentation on prompt caching mechanics and pricing (Anthropic, OpenAI, DeepSeek

  • [LB] Anthropic prompt caching hashes the request prefix in the fixed order tools -> system -> messages, requires 100% identical content up to the cache_control block, and changes at one level invalidate that level and everything after it.
  • [LB] On Anthropic, cache writes happen ONLY at explicit cache_control breakpoints (max 4 per request); on later requests the system walks backward at most 20 blocks from each breakpoint looking for previously WRITTEN entries — so a breakpoint placed on a block that changes every call (e.g., a running summary) produces zero cache hits; the breakpoint must sit on the last STABLE block.
    • Источник (official-doc): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.md — Anthropic Docs: Prompt caching
    • Каветы: This directly answers key question (1) for Anthropic: a changing running summary at the end leaves everything before it cacheable, but only if the breakpoint is on the last stable block, not auto-placed on the final (volatile) block. The doc's worked 'common mistake' example is exactly the changing-suffix case.
    • VERDICT: CONFIRMED
  • [LB/M] Anthropic caching pricing: 5-minute-TTL cache writes cost 1.25x base input price, 1-hour-TTL writes cost 2x, cache reads cost 0.1x; reading cached content refreshes its TTL at no additional cost.
  • [LB/M] Anthropic minimum cacheable prompt length is model-dependent (per the live doc: 512 tokens for Claude Fable 5; 1,024 for Opus 4.8/Sonnet 5/Sonnet 4.6/4.5; 2,048 for Opus 4.7; 4,096 for Opus 4.6/4.5 and Haiku 4.5); shorter prompts are silently processed without caching, with no error.
    • Источник (official-doc): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.md — Anthropic Docs: Prompt caching
    • Каветы: Verbatim table re-extracted twice with identical values, but this table visibly changes across model generations (an older cached copy of the same doc showed different per-model values) — treat the exact numbers as volatile and re-check before hardcoding; the design-stable fact is 'minimum is ~512–4,096 tokens and failure is silent — verify via usage.cache_read_input_tokens'.
    • VERDICT: CONFIRMED
  • [LB] An Anthropic cache entry becomes available only after the first response BEGINS; parallel requests with identical prefixes all pay full (write) price, so fan-out should wait for the first response before firing the rest. This also bounds retry economics: a retry of a request that failed before the response began pays the cache-write price again (no entry was written), while a retry after a mid-stream failure or completed-but-rejected generation reads the prefix at 0.1x.
    • Источник (official-doc): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.md — Anthropic Docs: Prompt caching
    • Каветы: The second sentence of the claim (retry billing split) is my inference from the quoted timing rule plus the read/write multipliers — no provider doc states error-path cache billing explicitly. Anthropic also never bills a 'retry' specially: the retry is an ordinary request billed as read(prefix)+input(suffix) or write+input.
    • VERDICT: OVERSTATED — коррекция: Per Anthropic's prompt-caching docs, a cache entry only becomes available after the first response begins, so concurrent requests with identical prefixes will not get cache hits from each other's in-flight writes; for cache hits on parallel fan-out, wait for the first response before sending subsequent requests. Cache reads bill at 0.1x base input price and 5-minute-TTL cache writes at 1.25x (1-hour: 2x). The retry-economics corollary — that a retry of a request failing before the response began re-pays the cache-write price, while a retry after a mid-stream failure or rejected generation reads the prefix at 0.1x — is an inference from this sentence, not something the documentation states.
  • [LB/M] OpenAI prompt caching is fully automatic (no code changes, no cache_control equivalent) on gpt-4o and newer, applies only to prompts of 1024+ tokens, and requires exact prefix matches; routing uses a hash of roughly the first 256 tokens, optionally steered with the prompt_cache_key parameter.
    • Источник (official-doc): https://developers.openai.com/api/docs/guides/prompt-caching — OpenAI Docs: Prompt caching
    • Каветы: Because there are no breakpoints, OpenAI answers question (1) implicitly: the cacheable prefix is simply the longest exact-matching prefix, so a changing summary at the end truncates the hit there automatically — no placement bug is possible, unlike Anthropic. Cached-token count is reported in usage prompt_tokens_details.cached_tokens.
  • [LB] OpenAI explicitly instructs the static-before-dynamic ordering: put instructions/examples at the beginning and variable, user-specific content at the end; images and tools must be byte-identical to count toward the prefix.
    • Источник (official-doc): https://developers.openai.com/api/docs/guides/prompt-caching — OpenAI Docs: Prompt caching
    • Каветы: This is the direct official answer to key question (2), and it matches Anthropic's and Gemini's guidance — three providers independently prescribe the same layout, so the ordering rule is provider-portable.
  • [LB/M] OpenAI cache economics (per live 2026 pricing page): cached input tokens are billed at 10% of the standard input rate (90% discount) across current models offering caching (e.g., gpt-5.4: $2.50 input / $0.25 cached); on pre-GPT-5.6 models cache writes are free, while GPT-5.6+ introduces 1.25x-priced cache writes and an optional prompt_cache_options.ttl of '30m'; default retention is 5–10 minutes of inactivity, up to 1 hour (24h on select extended-retention models).
    • Источник (official-doc): https://developers.openai.com/api/docs/pricing and https://developers.openai.com/api/docs/guides/prompt-caching — OpenAI Docs: Pricing; Prompt caching
    • Каветы: The 90% figure is computed from the pricing table (cached price / input price), consistent across listed models; some pro-tier models list no caching at all. Model names (gpt-5.4/5.5/5.6) are post-training-cutoff and taken as extracted from the live page. OpenAI docs say nothing about failed-request billing or whether an errored request populates the cache.
  • [LB] DeepSeek context caching on disk is enabled by default for all users with no code changes; a request is a cache hit only for prefixes identical from token 0; storage granularity is 64-token units (content under 64 tokens is never cached); hits/misses are reported per request as prompt_cache_hit_tokens / prompt_cache_miss_tokens; the cache is best-effort with no guaranteed hit rate, and unused entries are cleared 'within a few hours to a few days'.
    • Источник (official-doc): https://api-docs.deepseek.com/guides/kv_cache and https://api-docs.deepseek.com/news/news0802 — DeepSeek API Docs: Context Caching; DeepSeek API introduces Context Caching on Disk
    • Каветы: For question (1): a changing running summary near the end truncates the hit at the last identical 64-token boundary; everything before caches automatically. The kv_cache guide adds that cache units are carved at the end of user input and end of model output plus fixed token intervals, which is what makes multi-turn/next-chunk reuse work. Per-user cache isolation is explicit ('logically invisible to others').
  • [LB/M] DeepSeek current pricing (live page, July 2026): deepseek-v4-flash — $0.0028/M input on cache hit, $0.14/M on cache miss, $0.28/M output; deepseek-v4-pro — $0.003625/M hit, $0.435/M miss, $0.87/M output; legacy names deepseek-chat and deepseek-reasoner are deprecated on 2026-07-24 and map to v4-flash modes.
    • Источник (official-doc): https://api-docs.deepseek.com/quick_start/pricing — DeepSeek API Docs: Models & Pricing
    • Каветы: Hit prices imply a ~98–99% discount vs miss — far deeper than the historical 90% ($0.014 vs $0.14 at the 2024 launch). Extracted twice with identical digits, but both extractions came from one page fetch through a summarizing layer; re-verify these exact hit prices against the live page before baking them into a COGS model. The deprecation date is 8 days after today — model-slug migration is urgent for any DeepSeek-calling code.
  • [LB/M] Gemini implicit caching is enabled by default for all Gemini 2.5+ models with no guarantee of savings; minimum input for a hit is 2,048 tokens (2.5 Flash/Pro) or 4,096 tokens (3.x-generation models); Google's official guidance is to put large common content at the beginning of the prompt and to send similar-prefix requests close together in time; cached-token counts appear in usage metadata (cached_content_token_count / total_cached_tokens depending on API surface).
    • Источник (official-doc): https://ai.google.dev/gemini-api/docs/generate-content/caching and https://ai.google.dev/gemini-api/docs/caching — Google AI for Developers: Context caching
    • Каветы: Google's docs split by API surface in 2026: the new Interactions API supports implicit caching only, while the legacy Generate Content API supports both implicit and explicit. 'Send requests with similar prefix in a short amount of time' is the same warm-cache-sequentially advice as Anthropic's concurrent-request rule, stated softly.
  • [LB/M] Gemini explicit caching (caches.create / CachedContent on the Generate Content API) has a default TTL of 1 hour with no min/max bounds, carries a cost-saving GUARANTEE (unlike implicit), and bills three components: cached tokens at a reduced rate (~10% of standard input, e.g. 2.5 Pro $0.125/M cached vs $1.25/M input), storage per token-hour ($4.50/M-tok/hr for 2.5 Pro, $1.00/M-tok/hr for Flash-tier models), plus normal rates for non-cached input and output.
    • Источник (official-doc): https://ai.google.dev/gemini-api/docs/generate-content/caching and https://ai.google.dev/gemini-api/docs/pricing — Google AI for Developers: Context caching; Gemini Developer API pricing
    • Каветы: Storage is billed on wall-clock TTL duration regardless of use — unlike Anthropic (free refresh-on-use) and OpenAI/DeepSeek (no storage charge at all), so explicit Gemini caches only pay off for heavily reused static content (a book-wide system prompt + glossary fits; per-chunk content does not). CachedContent fields (model, contents, tools, systemInstruction) are immutable after creation.
  • [LB] No provider's official caching documentation states what is billed when a generation FAILS or is REJECTED and then retried; retry cost must be derived from the general rules: the retry is billed as a normal request, with the identical prefix billed at cache-read rates only if (a) the prior attempt progressed far enough to write the cache (Anthropic: response began; others: unspecified), (b) the retry arrives within the retention window (Anthropic 5m/1h; OpenAI 5–10m; DeepSeek best-effort hours-days; Gemini implicit unspecified / explicit TTL), and (c) the retry body is byte-identical.
    • Источник (official-doc): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.md (plus the OpenAI, DeepSeek, Gemini caching pages cited above) — Anthropic Docs: Prompt caching
    • Каветы: This is a synthesis finding: the quoted Anthropic sentence is the only official statement that constrains the failure/retry path; the rest is inference from documented mechanics, flagged as such. Practical consequence: a judge-rejected translation retried with the identical prompt is cheap (prefix at 0.1x/10%/hit-rate pricing), but a retry that MODIFIES the prompt (e.g., appending 'previous attempt was rejected because...') invalidates from the edit point onward.

t8-memory-state — Sequential whole-book processing with LLMs: recursive summarization error accumulation, running-summ

  • [LB/M] In recursive book summarization (GPT-3 175B, RLHF), errors made in lower-level summaries compound at each composition step: full-book Likert scores were significantly lower than scores of any individual decomposed subtask.
    • Источник (paper): https://arxiv.org/abs/2109.10862 — Wu et al. 2021, "Recursively Summarizing Books with Human Feedback" (OpenAI)
    • Каветы: 2021 GPT-3-era models; absolute magnitudes are likely better with modern models, but the structural point (hierarchical composition propagates lower-level errors upward) is architecture-level, not model-level.
    • VERDICT: CONFIRMED
  • [ /M] Even the best recursive-summarization model produced human-comparable book summaries rarely: ~5% of books rated 6/7 and >15% rated 5/7, significantly below human-written summaries on average.
    • Источник (paper): https://arxiv.org/abs/2109.10862 — Wu et al. 2021, "Recursively Summarizing Books with Human Feedback"
    • Каветы: Calibration data point for 2021-era models only; not a statement about current models.
  • [LB/M] Incrementally updating a running summary chunk-by-chunk produces measurably less coherent book summaries than hierarchical chunk-summarize-then-merge, across models: GPT-4 82.5 vs 89.1 BooookScore, GPT-3.5-Turbo 67.0 vs 84.2, Mixtral-8x7B 64.5 vs 81.5; the gap closes only with very large chunks (Claude 2 at 88K-token chunks: 90.9 incremental vs 90.3 hierarchical).
    • Источник (paper): https://arxiv.org/abs/2310.00785 — Chang et al., "BooookScore: A systematic exploration of book-length summarization in the era of LLMs" (ICLR 2024)
    • Каветы: BooookScore measures coherence of the summary text (1193 human annotations on 100 books, 8 error types), NOT faithfulness to the source book; incremental updating retained more detail. 2023/24 models.
    • VERDICT: CONFIRMED
  • [LB/M] Omission is the dominant error class in book-length summarization, and running-summary (incremental) updating roughly doubles per-sentence error rates vs hierarchical merging: entity omission 7.3% vs 3.71%, event omission 4.25% vs 2.27%, causal omission 2.75% vs 1.21%, duplication 2.12% vs 1.18%.
    • Источник (paper): https://arxiv.org/html/2310.00785v4 — BooookScore (ICLR 2024), Table 1
    • Каветы: Rates are per-sentence over GPT-4 summaries of 100 books; 'compounding omission' is the endpoint measurement — the paper does not plot error rate as a function of update count.
    • VERDICT: CONFIRMED
  • [LB/M] Recursive merging of summaries without source access amplifies hallucination; re-grounding merge steps with extracted source context measurably improves faithfulness: manual claim-accuracy 72.7% vs 59.1% for vanilla hierarchical merging, and AlignScore 85.8 vs 76.3 (Llama-3.1-70B, Multi-LexSum).
    • Источник (paper): https://arxiv.org/abs/2502.00977 — Ou & Lapata 2025, "Context-Aware Hierarchical Merging for Long Document Summarization"
    • Каветы: Manual evaluation was on one book only; automatic faithfulness metrics (AlignScore/SummaC) are themselves noisy. Tested on legal/gov/book summarization, not translation.
    • VERDICT: OVERSTATED — коррекция: Recursive merging of summaries can amplify LLM hallucinations; enriching merge steps with source-document context measurably improves faithfulness (Ou & Lapata 2025). On Multi-LexSum with Llama-3.1-70B, all context-aware variants beat vanilla hierarchical merging (AlignScore 76.3): citation-based Cite-Replace is best at 85.8, while extraction-based variants reach 79.0–79.9. Separately, a small manual claim-accuracy study on one SuperSummary book (Llama-3.1-70B) found 72.7% correct claims for Extract-Support vs 59.1% for vanilla hierarchical merging.
  • [LB/M] OS-style paged memory with structured, agent-editable state (MemGPT: core memory blocks + append-only recall storage + archival store) held consistency where fixed-context baselines collapsed: deep memory retrieval accuracy 92.5% vs 32.1% (GPT-4), 93.4% vs 35.3% (GPT-4 Turbo), and nested key-value multi-hop lookup where fixed-context baselines hit 0% at 3+ nesting levels.
    • Источник (paper): https://arxiv.org/abs/2310.08560 — Packer et al. 2023, "MemGPT: Towards LLMs as Operating Systems"
    • Каветы: Evaluated on multi-session dialogue and document QA, not translation or narrative generation; 2023 GPT-3.5/GPT-4. MemGPT's memory paging is LLM-driven, not deterministic — later work notes its naive summary-merge itself drifts. Letta docs page (docs.letta.com/memory/overview) returned 404, so current vendor framing unverified.
    • VERDICT: CONFIRMED
  • [LB/M] Memory representation should match query type: free-prose summaries and raw chunks win on long-context comprehension (summaries best on NarrativeQA F1 32.93%, chunks best on QuALITY 78.5%), while structured atomic facts/knowledge triples win on precise relational lookup, and mixed memory is the most noise-robust (F1 82.11% on HotpotQA).
    • Источник (paper): https://arxiv.org/abs/2412.15266 — Zeng et al. 2024, "On the Structural Memory of LLM Agents"
    • Каветы: QA/dialogue benchmarks, not generation or translation; 'summary vs structured' trade-off measured for retrieval accuracy, not for state-update drift.
  • [LB/M] A deterministic bilingual term record (dictionary of each proper noun's FIRST translation, injected as a constraint into every subsequent sentence prompt) is a measured fix for name/term drift: DelTA raises lexical translation consistency (LTCR-1) by up to 4.58pp on main test sets and from 37.00 to 85.50 (+48.5pp) on the GuoFeng webnovel set, while also raising COMET by up to 3.16.
    • Источник (paper): https://arxiv.org/abs/2410.08143 — Wang et al., "DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory" (ICLR 2025)
    • Каветы: LTCR-1 defines consistency as matching the FIRST translation — it rewards lock-in, not correctness of the first choice; a wrong first translation is propagated (summary-poisoning analog). En↔Zh/De etc., not →ru. Same broader author line (Longyue Wang / Tencent) as the GuoFeng corpus — partially one voice with the BWB/doc-MT findings.
  • [LB/M] Window/chunk-based whole-document LLM translation loses sentences as windows grow: with a 50-sentence window the baseline dropped 10 target sentences despite complete source input; DelTA's sentence-by-sentence strategy with memory avoided omissions entirely.
    • Источник (paper): https://arxiv.org/html/2410.08143v1 — DelTA (ICLR 2025)
    • Каветы: Small-scale illustration within one paper; exact omission-vs-window-size curve not published. Sentence-by-sentence translation trades throughput and paragraph-level fluency for coverage.
  • [LB/M] Giving the translator paragraph-level context sharply reduces consistency errors vs sentence-by-sentence translation: 2 vs 25 inconsistency errors, 480 vs 622 mistranslations, 102 vs 156 grammar errors (GPT-3.5, 18 language pairs incl. zh/ja/ru sources), and translators preferred paragraph-level 71.1% of the time — yet critical errors persist and paragraph mode occasionally omits content.
    • Источник (paper): https://arxiv.org/abs/2304.03245 — Karpinska & Iyyer 2023, "Large language models effectively leverage document-level context for literary translation, but critical errors persist"
    • Каветы: GPT-3.5 (2023); context = one paragraph, not cross-chapter state, so it does not measure book-scale consistency; error counts are totals over the study corpus, not rates per length.
  • [LB/M] In Chinese web-novel document-level MT specifically, named entities and terminology are the dominant document-level failure: 43.3% of document-level errors were named-entity mistranslations and 19.2% terminology mistakes; document-level MT coreference F1 was 63.5 vs 88.1 for humans (B3).
    • Источник (paper): https://arxiv.org/abs/2305.11142 — "Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus" (BWB corpus, 6 web novels)
    • Каветы: zh→en web novels — closest published genre match to ranobe/webnovel work, but not →ru; pre-LLM-era NMT systems in some comparisons. Overlaps in authorship with the Tencent doc-MT line — count as partially one voice with DelTA/GuoFeng.
  • [ /M] The append-only event log + periodic synthesized reflection pattern is measurably better than reflection-free memory: full architecture TrueSkill mu=29.89 vs 26.88 without reflection vs 21.21 with observations/reflection/planning all removed (effect size d=8.16 vs prior-work baseline).
    • Источник (paper): https://arxiv.org/abs/2304.03442 — Park et al. 2023, "Generative Agents: Interactive Simulacra of Human Behavior"
    • Каветы: Measured on believability of simulated social agents — far from translation; supports the append-only-log + periodic-consolidation pattern only at the architecture level.
  • [ /M] Reported claims that agent memory drifts catastrophically over hundreds of interactions (e.g., 42% task-success reduction beyond ~200-300 interactions; iterative lossy re-summarization progressively distorting stored facts) circulate in 2026 preprints/surveys but I could only verify them via secondary aggregators, not primary texts.
    • Источник (blog): https://www.emergentmind.com/papers/2601.04170 — "Quantifying Agent Drift in Multi-Agent LLMs" (aggregator summary); https://arxiv.org/pdf/2603.11768 — SSGM framework preprint
    • Каветы: UNVERIFIED against the primary paper; treat as directional only. The qualitative mechanism (repeated lossy re-encoding distorts state) is consistent with the verified BooookScore and Wu et al. findings.

добор: g1-cjk-sentence-primitives

  • [LB] Unicode UAX #29 defines sentence boundaries with 。！？ in the STerm class (ATerm holds only dot-like characters: U+002E, U+2024, U+FE52, U+FF0E), and rules SB9–SB11 keep closing quotes/brackets after a terminator attached to the sentence, so the break in 「…。」 falls AFTER the closing quote, not after 。.
    • Источник (standard): https://www.unicode.org/reports/tr29/#Sentence_Boundaries — UAX #29: Unicode Text Segmentation, Sentence Boundaries
    • Каветы: This is exactly the case naive 'split after 。！？' code gets wrong; a correct deterministic splitter must consume the Close* run (」』）” etc.) before emitting the boundary.
    • VERDICT: CONFIRMED
  • [LB] UAX #29's default algorithm produces no sentence break at all when there is no terminal punctuation (fallback rule SB998 'Any × Any'), and the standard explicitly documents that plain text is inadequate for good boundaries (abbreviations, ellipsis, decimal periods), recommending tailored implementations.
    • Источник (standard): https://www.unicode.org/reports/tr29/#Sentence_Boundaries — UAX #29: Unicode Text Segmentation
    • Каветы: For webnovel lines that end without punctuation, the only deterministic boundary the standard gives you is the paragraph/line separator — the chunker must treat newline as a boundary of last resort by design, not hope the splitter finds one.
    • VERDICT: OVERSTATED — коррекция: Within a paragraph, UAX #29's default sentence-boundary rules produce no sentence break in the absence of sentence-terminating punctuation — the fallback rule SB998 "Any × Any" ("Otherwise, do not break") applies — though SB4 still breaks after paragraph separators (Sep | CR | LF). The standard explicitly states that "plain text provides inadequate information for determining good sentence boundaries," citing periods' ambiguity between end-of-sentence, abbreviations, and decimal points (ellipsis is not mentioned), and says implementations "can and should tailor" the default rules for particular locales or environments.
  • [LB] ICU's sentence BreakIterator implements the UAX #29 rules and offers abbreviation-suppression tailoring (locale tag ss=standard, ICU 64+) only for English, German, Spanish, French, Italian and Portuguese — there is no CJK-specific sentence tailoring, and ICU states more sophisticated linguistic analysis 'is not directly available from ICU at this time'.
    • Источник (official-doc): https://unicode-org.github.io/icu/userguide/boundaryanalysis/ — ICU User Guide: Boundary Analysis
    • Каветы: Means an ICU dependency buys you nothing over a faithful UAX #29 port for zh/ja; the CJK edge cases must be handled by your own tailoring layer either way.
    • VERDICT: CONFIRMED
  • [LB] A pure-Go UAX #29 implementation with a sentence segmenter exists: github.com/clipperhouse/uax29 splits graphemes/words/sentences per Unicode 17 and is validated against the official Unicode test suites — a viable deterministic base layer without cgo/ICU bindings.
    • Источник (practitioner): https://github.com/clipperhouse/uax29 — clipperhouse/uax29: A tokenizer based on Unicode text segmentation (UAX #29), for Go
    • Каветы: Passing the Unicode conformance suite ≠ good behavior on webnovel text (emoji/kaomoji/no-punctuation lines are outside the suite); a tailoring layer on top is still required. The popular alternative neurosnap/sentences is a punkt port with no CJK support (https://github.com/neurosnap/sentences).
  • [LB/M] Measured Japanese benchmark (hkiyomaru/ja-senter-benchmark, micro-F1): rule-based splitters score 94.8–100 on Wikipedia and 83.7–88.4 on CC-100 web text, but 0.0 F1 on emoji-boundary and kaomoji test sets (pysbd, rhoknp, kuzukiri, hasami) and 44.4 on line-break cases; bunkai scores 100 on emoji and line breaks but 0 on named-entity periods (モーニング娘。); no tool wins everywhere.
    • Источник (practitioner): https://github.com/hkiyomaru/ja-senter-benchmark — ja-senter-benchmark: comparison of Japanese sentence segmentation tools
    • Каветы: Small test sets (15 Wikipedia docs, 15 CC-100 texts, hand-made edge-case sets annotated by the repo owner); authored by a researcher but not peer-reviewed. Directly relevant to webnovel style: the failure classes are exactly emoji/kaomoji/line-break cases.
  • [ ] Bunkai (Megagon Labs) is the documented reference design for Japanese UGC sentence splitting: a rule-based annotator pipeline handling emoji/emoticon terminators (♪★(笑)), plus an OPTIONAL ML component only for line-break disambiguation — i.e., the deterministic subset covers punctuation variety, and only 'is this newline a boundary?' was deemed to need a model (Hayashibe & Mitsuzawa, W-NUT 2020).
    • Источник (paper): https://github.com/megagonlabs/bunkai — megagonlabs/bunkai; paper: https://aclanthology.org/2020.wnut-1.10/ — Sentence Boundary Detection on Line Breaks in Japanese
    • Каветы: Paper PDF F1 numbers not extracted (parse failure); the ja-senter-benchmark row for bunkai serves as the measured proxy. For a chunker, treating ambiguous newlines conservatively (prefer NOT to end a sentence, but allow chunk break at paragraph marks) sidesteps the ML need.
  • [ /M] pragmatic_segmenter (rule-based, Ruby, the origin of the 'Golden Rules' test set) has dedicated rules for Japanese and Chinese; its Japanese golden rules keep text inside 「」 quotes unsplit (e.g. 「引き下げ幅は…」 stays one segment) and handle errant mid-sentence newlines; self-reported Golden Rules accuracy is 98.08% (English) and 100% (other languages), versus 31–66% for Stanford CoreNLP/OpenNLP/TactfulTokenizer.
    • Источник (practitioner): https://github.com/diasks2/pragmatic_segmenter — diasks2/pragmatic_segmenter
    • Каветы: Self-reported benchmark on the author's own test set (circular); the Golden Rules themselves are still the best available portable edge-case checklist for zh/ja/en, and ja_sentence_segmenter (https://github.com/wwwcojp/ja_sentence_segmenter) ports the JA subset to a split-then-reconcatenate pipeline (split_punctuation on 。!? + concatenate_matching for quotes/particles).
  • [ ] Portable SRX rule sets covering CJK exist and are inspectable: LanguageTool ships all-language rules in one segment.srx (Japanese break rule pattern reported as [\xff61\x3002\xff0e\xff1f\xff01]+ — halfwidth ideographic stop, 。, fullwidth ．？！), and OmegaT ships defaultRules.srx with a Japanese rule breaking after 。 with empty after-context; SRX groups per-language break/exception regex rules with cascading.
  • [LB] No Go implementation of SRX was found; the reference open implementations are Java (loomchild/segment, used by LanguageTool) and Rust (bminixhofer/srx, 'mostly compliant' SRX 2.0), and the standard itself specifies ICU regular expressions, which are documented as hard to reproduce exactly in other regex engines.
    • Источник (practitioner): https://github.com/bminixhofer/srx — bminixhofer/srx (Rust SRX 2.0); https://okapiframework.org/wiki/index.php?title=SRX — Okapi Framework: SRX
    • Каветы: Implication: consuming .srx files directly in Go means writing/porting an SRX engine and dealing with ICU-vs-RE2/regexp semantics (lookbehind is not in Go's regexp); hand-porting the small CJK rule set into native Go code is the lower-risk path.
  • [LB/M] For splitting oversized Chinese sentences, the comma is the documented clause-boundary device: Jin et al. (SIGHAN 2004) segment long Chinese sentences at classified commas with 87.1% comma-classification accuracy (improving parser accuracy by 9.6%), and Xue & Yang (ACL 2011) treat Chinese sentence segmentation as comma classification, since a subset of commas function as true end-of-sentence.
    • Источник (paper): https://aclanthology.org/W04-1101/ — Segmentation of Chinese Long Sentences Using Commas (Jin, Kim, Kim, Lee, SIGHAN 2004); https://aclanthology.org/P11-2111/ — Chinese sentence segmentation as comma classification (Xue & Yang, ACL 2011)
    • Каветы: Both use trained classifiers, not deterministic rules — 87% is the ceiling WITH ML; a deterministic 'any ，、；：' split is safe for token-budget purposes only if such cuts are marked soft/second-class (they are clause, not sentence, boundaries). Xue & Yang's EOS-comma F1 was reported around 70 via search snippet — not verified against the PDF. Related follow-up: Chinese Comma Disambiguation for Discourse Analysis (https://aclanthology.org/P12-1083.pdf).
  • [LB/M] DeepSeek officially publishes both a chars-per-token heuristic and an offline tokenizer: '1 English character ≈ 0.3 token, 1 Chinese character ≈ 0.6 token' (i.e., ≈3.3 en chars/token, ≈1.67 zh chars/token), with a downloadable deepseek_tokenizer.zip for computing exact usage locally without API calls, and an explicit caveat that the heuristic varies by model.
    • Источник (official-doc): https://api-docs.deepseek.com/quick_start/token_usage — DeepSeek API Docs: Token & Token Usage
    • Каветы: Independent measurement converges: DeepSeek R1 measured at 1.54 chars/token on C4 Chinese (see Medium tokenizer comparison), close to the official 1.67. No official ratio is published for Russian or Japanese — measure locally with the offline tokenizer. DeepSeek-V3 uses byte-level BPE, 128K vocab (https://arxiv.org/html/2412.19437v1 — DeepSeek-V3 Technical Report).
  • [LB] GLM-4's tokenizer is offline-computable in the same rank format tiktoken uses: the HF tokenization_chatglm.py imports tiktoken and loads base64-encoded mergeable ranks from a local tokenizer.model with a fixed pretokenizer regex (fully offline); per the GLM-4 report the vocabulary is 150K, byte-level BPE merged with cl100k_base tokens, with ~28K Chinese tokens.
    • Источник (official-doc): https://huggingface.co/THUDM/glm-4-9b/blob/main/tokenization_chatglm.py — THUDM/glm-4-9b tokenization_chatglm.py; https://arxiv.org/pdf/2406.12793 — ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
    • Каветы: Because the format matches tiktoken's (base64 token + rank per line), Go tiktoken libraries with pluggable BPE loaders (e.g. pkoukk/tiktoken-go SetBpeLoader) can load GLM's vocabulary — exact GLM token counts in pure Go. Verify the newest GLM (glm-4.x/5 candidate) still ships the same tokenizer file before relying on it.
  • [LB] Pure-Go tiktoken implementations exist with embedded vocabularies (no runtime downloads): tiktoken-go/tokenizer embeds OpenAI vocabularies (~4MB) compiled into the binary and supports o200k_base and cl100k_base; pkoukk/tiktoken-go offers an offline BPE loader and pluggable loaders for custom rank files.
  • [LB/M] Gemini's official guidance is 'a token is equivalent to about 4 characters' for Gemini models, with exact counts available via the countTokens API; local (non-API) counting exists via SentencePiece-based tokenizers: Python vertexai.preview.tokenization and a Go package cloud.google.com/go/vertexai/genai/tokenizer that counts locally but downloads its tokenizer model from the web at runtime.
  • [ /M] Measured on >2M translated sentence pairs under cl100k_base: Mandarin Chinese averages 1.76x the tokens of the English equivalent, Japanese 2.12x (with outlier sentences up to 8x), Korean 2.36x; rare hanzi can cost 2–3 tokens per character while common ones cost 1.
    • Источник (blog): https://tonybaloney.github.io/posts/cjk-chinese-japanese-korean-llm-ai-best-practices.html — Working with Chinese, Japanese, and Korean text in Generative AI pipelines (Anthony Shaw)
    • Каветы: Practitioner blog (Microsoft engineer), cl100k only — pre-o200k, so numbers are stale for GPT-4o-class models but remain the largest-corpus CJK measurement found. Also recommends splitting CJK on fullwidth 。．！？ rather than ASCII '.'.
  • [LB/M] Measured six-task zh/ja corpus comparing OpenAI tokenizers: Chinese token-ratio vs English fell from 2.08x (cl100k_base) to 1.34x (o200k_base), Japanese from 2.32x to 1.73x; English runs ~5.4 chars/token vs Chinese 0.9–1.0 chars/token, and the authors warn vendor ratios cannot be inferred from tiktoken.
    • Источник (blog): https://masonailab.com/en/insights/token-efficiency/ — CJK Token Myth Busted — Measured Data | Mason AI Lab
    • Каветы: Small corpus (six task types, not a large-scale study); marketing-adjacent blog. Load-bearing because the safety-margin choice depends on this spread (chars/token for zh varies ~1.0 → ~1.7 across tokenizers, i.e. up to ~70% divergence if one table is applied to another vendor).
  • [LB/M] Same-text measurement (1,403-char Chinese article, translated variants): Gemini 3 Flash used 2,141 tokens for Japanese where DeepSeek V3.2 used 856 (2.5x), and 1,631 vs 636 for English; measured chars/token was 0.82–1.84 for Gemini vs 2.02–4.16 for DeepSeek — i.e., Gemini's SentencePiece (~256K vocab) is 2–2.8x less token-efficient per character than DeepSeek's BPE on this text.
  • [ /M] Russian tokenization is materially worse than English under GPT tokenizers but precise ratios are scarce: a Wikipedia-corpus cl100k study reports Russian costing 'more than double' English (chart-only, no table), and Petrov et al. (2023) measured cross-language tokenization length differences up to 15x across 17 tokenizers, establishing that per-language premiums are tokenizer-specific and large.
    • Источник (paper): https://tomaszurbanski.substack.com/p/the-hidden-price-tag-on-gpt-4-for — GPT-4's Hidden Cost: Is Your Language Pricing You Out (Substack); https://aleksandarpetrov.github.io/tokenization-fairness/ — Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al., 2023)
    • Каветы: No source provided an exact chars/token for Russian under DeepSeek/GLM/Gemini — and Russian is your OUTPUT language, so output-budget math needs a locally-measured number (trivial to obtain with the offline tokenizers above). A secondary blog claims 3–4 tokens per Cyrillic word under cl100k (unverified).
  • [ ] DeepSeek-V3's pretokenizer merges punctuation with line breaks into combined tokens (a compression optimization), which the technical report itself flags as introducing token-boundary bias around line-break-adjacent text — meaning chars/token near paragraph boundaries differs from mid-paragraph text.
    • Источник (paper): https://arxiv.org/html/2412.19437v1 — DeepSeek-V3 Technical Report
    • Каветы: Minor for budgeting (affects a few tokens per paragraph) but a concrete reason why chars-per-token heuristics drift with text shape (dialogue-heavy webnovel text has many short lines); exact offline tokenization absorbs this automatically.

добор: g2-diff-editor-contract

  • [LB/M] In the WMT 2019 APE shared task English-Russian round, NO submitted system improved over the do-nothing baseline (leaving the strong NMT output untouched, 16.16 TER / 76.20 BLEU); every APE run degraded or failed to beat it.
    • Источник (paper): https://aclanthology.org/W19-5402/ — Findings of the WMT 2019 Shared Task on Automatic Post-Editing (Chatterjee et al.)
    • Каветы: Pre-LLM APE systems (2019, transformer-era) on news domain; only 2 teams / 4 runs on en-ru. Still the single most direct evidence that on a strong draft into Russian, an unconstrained post-editor's expected value is negative — the editor contract needs an explicit, cheap do-nothing path and must be benchmarked against do-nothing.
    • VERDICT: CONFIRMED
  • [LB/M] WMT APE gains collapse as base-MT quality rises: PBSMT-era drafts allowed up to -6.24 TER / +9.53 BLEU improvement (WMT 2018), but on NMT drafts the best 2019 en-de system managed only -0.78 TER / +1.23 BLEU, and organizers attribute the difficulty to over-correction of already-adequate output.
    • Источник (paper): https://www.aclweb.org/anthology/W18-6452 and https://aclanthology.org/W19-5402/ — Findings of the WMT 2018/2019 Shared Tasks on Automatic Post-Editing
    • Каветы: The WMT findings series (2018-2023) is one organizer voice across years, though datasets differ. Numbers are corpus-level TER/BLEU, not a per-sentence 'X% of adequate sentences degraded' rate — that finer statistic was not extractable from abstracts fetched. Later rounds (2020 en-de: -11.35 TER; en-zh: -12.13 TER) used deliberately lower-quality drafts, so large gains there do not contradict the trend.
    • VERDICT: CONFIRMED
  • [ /M] Over-correction is an explicitly named, still-open APE failure mode in 2025: unnecessary modifications violating minimal-editing; injecting word-level QE 'OK/BAD' tags as decoding constraints (preserve OK tokens) improves APE by 0.65 TER (en-de), 1.86 (en-hi), 1.44 (en-mr) over unconstrained APE.
    • Источник (paper): https://arxiv.org/abs/2501.17265 — Giving the Old a Fresh Spin: Quality Estimation-Assisted Constrained Decoding for Automatic Post-Editing (NAACL 2025)
    • Каветы: Token-level constrained decoding is not directly available through commercial chat APIs; the transferable idea is gating which spans the editor may touch. No Russian pair tested.
  • [LB/M] GPT-4 as a translation post-editor (WMT-22 test sets) improved COMET-22 on all pairs tested (e.g., zh-en 80.35→82.79, en-zh 86.69→87.53, en-de 86.78→87.37), and a chain-of-thought 'propose edits, then apply' format cut edit magnitude by roughly a third (TER between draft and post-edit: en-zh 94.5→70.3, zh-en 34.9→22.0) while keeping the quality gain; authors still document hallucinated edits and urge caution.
    • Источник (paper): https://arxiv.org/abs/2305.14878 — Leveraging GPT-4 for Automatic Translation Post-Editing (Raunak et al., EMNLP 2023 Findings)
    • Каветы: 2023, GPT-4 (frontier tier), sentence-level news/general domain, no Russian target, reference-based metrics. Key transferable result: an enumerate-edits-first output contract measurably reduces how much of the draft gets rewritten without losing the quality gain — direct support for the narrow-mandate design, but at frontier model class.
    • VERDICT: OVERSTATED — коррекция: GPT-4 post-editing of MS-Translator outputs on WMT-22 test sets improved COMET-22 on the four main pairs tested (zh-en 80.35→82.79, en-zh 86.11→87.53, en-de 86.78→87.37, de-en 84.68→85.41), with direct post-editing also surpassing the WMT-Best systems (e.g., en-zh WMT-Best 86.69); of twelve additional pairs tested in the CoT setting, most improved but En-Cs, En-Is, and Is-En declined. The chain-of-thought 'propose edits, then apply' format made edits substantially more conservative (TER between draft and post-edit: en-zh 94.5→70.3, zh-en 34.9→22.0, i.e., roughly a quarter to a third less editing), still improving over the initial translations though with a smaller quality gain than direct post-editing; the authors document hallucinated edits and urge caution.
  • [ /M] LLMs doing prose correction over-edit measurably: on CoNLL-14 GEC, ChatGPT (3-shot CoT) scored precision 51.3 vs GECToR 75.6 and T5-large 72.2 (F0.5 53.2 vs 66.3/66.8), and human evaluation rated it highest in over-correction among all compared systems; it ignores minimal-edit instructions most on short sentences.
    • Источник (paper): https://arxiv.org/abs/2304.01746 — Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation (2023)
    • Каветы: GEC on English learner text, GPT-3.5-era ChatGPT; precision penalty partially reflects reference bias of minimal-edit benchmarks (fluent rewrites scored as errors). Direction (LLM default = fluency rewrite, not minimal edit) is corroborated independently by the minimal-edit GEC line (arXiv 2506.13148) and edit-level majority-voting work.
  • [ ] Mitigations for LLM over-correction exist as a 2025-2026 research line: sampling multiple candidate corrections and keeping only edits that win an edit-level majority vote (arXiv 2605.13624, nine benchmarks, seven languages incl. Czech/Ukrainian/German), and error-rate-adapted training schedules for minimal-edit GEC (arXiv 2506.13148, SOTA single-model BEA-test).
    • Источник (paper): https://arxiv.org/pdf/2605.13624 and https://arxiv.org/abs/2506.13148
    • Каветы: Both are benchmark-backed papers but I could not extract their exact deltas from the abstracts fetched — treat magnitudes as unverified; the existence and mechanism (vote at the level of individual edits, not whole outputs — only possible if the editor emits discrete edits) is what matters for the design.
  • [LB/M] Aider's refactor benchmark (89 large Python refactor tasks): GPT-4 Turbo scored 20% with search/replace blocks vs 61% with unified diff, and lazy placeholder-comment tasks dropped 12→4; GPT-4-0613 similarly 26%→59%.
    • Источник (practitioner): https://aider.chat/docs/unified-diffs.html — Unified diffs make GPT-4 Turbo 3X less lazy (aider, Dec 2023)
    • Каветы: One practitioner voice (aider/Paul Gauthier) — all aider numbers in this report share that origin. Task = long code refactors where the failure mode is truncation/laziness, not fidelity; result is specific to GPT-4 Turbo 2023 and does NOT generalize to weaker models (see GPT-3.5 finding) nor to prose. Later aider defaults moved most models to search/replace ('diff') format.
  • [LB/M] Apply-side tolerance dominates end-to-end edit reliability: in aider's ablations, disabling flexible (fuzzy, context-anchored) patch application caused a 9X increase in editing errors, and dropping 'high level diff' prompting (encouraging whole-block replacement over minimal ± lines) added 30-50% more errors; aider also explicitly avoids line numbers in diffs.
    • Источник (practitioner): https://aider.chat/docs/unified-diffs.html — aider unified-diffs writeup
    • Каветы: Same single aider origin; code domain. Directly transferable: the deterministic Go applier should locate edit anchors tolerantly (whitespace/normalization-insensitive search, nearby-context matching) rather than requiring byte-exact patches, and the format should ask for coherent replaced spans, not line-number-addressed minimal diffs.
  • [LB/M] Weak/cheap models cannot emit edit formats reliably and do better returning the whole text: GPT-3.5-turbo scored 46% (whole-file) vs 30% (search/replace diff) on aider's original benchmark, with pathological behavior like putting the entire file in the ORIGINAL block; aider therefore assigns 'whole' format to weak models permanently.
    • Источник (practitioner): https://aider.chat/docs/benchmarks.html — GPT code editing benchmarks (aider, 2023)
    • Каветы: Same aider origin; 2023 models. Defines the capability floor: the narrow-mandate contract presumes the editor model clears a format-emission threshold — must be verified per candidate model, not assumed.
  • [LB/M] Current mid-tier models clear that threshold on aider's polyglot leaderboard 'percent using correct edit format': DeepSeek-V3.2 chat 98.2% (diff), Gemini 2.5 Flash 95.6% (diff), Qwen3-235B 92.9% (diff), Claude 3.5 Haiku 91.1% (diff); frontier models reach 97-100%; GPT-4o-mini is still run in whole-file mode.
    • Источник (practitioner): https://aider.chat/docs/leaderboards/ — Aider polyglot leaderboard
    • Каветы: Same aider origin; aider's 'diff' is fenced SEARCH/REPLACE blocks, not unified diff; compliance is measured on code with retry loops available; a 2-9% malformed-edit rate at mid-tier means the pipeline needs a validate-and-retry (or reject-edit) path. GPT-4o-mini's 100% is trivial — whole-file format has almost nothing to get wrong.
  • [LB/M] Diff-XYZ (JetBrains, arXiv Oct 2025; 1,000 real commits): for the model-GENERATES-the-diff task, search/replace is by far the most reliably emitted format (exact-match: GPT-4.1 0.95, Claude 4 Sonnet 0.94, Qwen 32B 0.68) vs unified diff (0.77 / 0.82 / 0.23); for the model-APPLIES-the-diff task the ranking flips (udiff apply EM: Claude 4 Sonnet 0.95, GPT-4.1 0.90, Qwen 7B 0.58, Qwen 3B 0.36, while search-replace apply plateaus ~0.57); verbose tagged udiff-l is best for the smallest models, and formatting choice barely helps models <7B.
    • Источник (paper): https://arxiv.org/abs/2510.12487 — Diff-XYZ: A Benchmark for Evaluating Diff Understanding (JetBrains Research, 2025)
    • Каветы: Code only (Python/JS/Java/Kotlin/Rust), single paper. Crucial reading for this pipeline: our applier is deterministic Go, so only the GENERATION numbers bind — and they favor search/replace emission at every tested size; the poor search-replace APPLY score reflects LLM-as-applier ambiguity, which a deterministic exact/fuzzy matcher avoids. The flat 0.57 apply score across models is suspicious (likely a format-ambiguity ceiling) and unexplained in what I extracted.
  • [ /M] Copy-as-Decode (arXiv 2604.18170, Apr 2026) shows grammar-constrained span-edit output ( + ) is mechanically lossless — 482/482 oracle cases round-trip byte-exactly, including a prose dataset (ProbeEdit: meeting notes, design doc, scientific abstract) — but SPAN SELECTION is the hard part for small models: a trained Qwen2.5-1.5B selector reached only 12-17% held-out exact match; copying via parallel prefill is 6.8-303x faster than regenerating tokens.
    • Источник (paper): https://arxiv.org/abs/2604.18170 — Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing (2026)
    • Каветы: Very recent preprint, tiny datasets (154 prose cases), mechanism-level study, selector results explicitly below deployment threshold. It is the only source found that evaluates structured span-edits on prose at all, and it says deciding WHERE to edit, not encoding the edit, is the bottleneck for small models.
  • [ /M] Cascaded editing — a strong model emits a concise edit sketch, a fine-tuned 14B model materializes it — beats the strong model editing directly: +11.1% Pass@2 over DeepSeek R1 alone on the Aider benchmark, with 13% less time and 19% less cost (FSE 2026).
    • Источник (paper): https://arxiv.org/abs/2604.19201 — Cascaded Code Editing: Large-Small Model Collaboration (FSE 2026)
    • Каветы: Code editing, single recent paper, requires a fine-tuned applier model. Supports the general pattern of separating edit-decision from edit-serialization, which the translator→editor pipeline mirrors (editor decides, Go code applies).
  • [ ] Both major vendors publish an official trained edit format and both avoid line numbers: OpenAI's GPT-4.1 guide open-sources the V4A apply_patch diff (context-anchored, 'no need for line numbers', model 'extensively trained' on it); Anthropic's official file-editing tool is str_replace-based (exact-match search/replace of a unique old string).
  • [LB/M] The closest published analogue to the narrow-mandate translation editor: guiding LLaMA-2 7B/13B to post-edit MT with external MQM error-span annotations (zh-en, en-de, en-ru) improves TER/BLEU/COMET; fine-grained span feedback beats generic 'improve this' feedback only after fine-tuning (4-6 BLEU gap); when fed 200 already-correct translations the models still edited them, causing a small automatic-metric drop.
    • Источник (paper): https://aclanthology.org/2024.findings-naacl.265/ (arXiv 2404.07851) — Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations (Ki & Carpuat, NAACL 2024 Findings)
    • Каветы: Sentence-level news MT, small open models, en-ru is INTO Russian only as one of three pairs; 'small drop on correct inputs' shows even span-guided editors do not fully abstain — an external do-nothing gate is still needed. No comparison of span-REPLACEMENT output vs free rewrite output was run; the model still regenerated full sentences.
  • [ ] Flag-only editing exists in evaluator form and is measurable: MQM-APE improves LLM error-span annotation quality by testing each flagged error via a post-edit and keeping only quality-improving edits, working consistently across eight LLMs including low-resource languages — i.e., an edit-verification loop can filter non-impactful edits without retraining.
    • Источник (paper): https://arxiv.org/abs/2409.14335 — MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators (2024)
    • Каветы: Aimed at evaluation (error annotation quality), not final translation quality; exact deltas not extracted from the fetched material. Demonstrates a per-edit accept/reject verifier pattern that maps directly onto a judge-gated narrow editor.

добор: g3-experiment-methodology

  • [LB/M] Miller (Anthropic) gives five concrete recommendations for LLM eval statistics: compute SEs via CLT, use clustered standard errors when questions are drawn in related groups, reduce variance by resampling (K generations per question), conduct inference on question-level PAIRED differences rather than aggregate scores, and run power analysis to size the eval. The paired-difference SE is SE_paired = sqrt(SE_A^2 + SE_B^2 - 2SE_ASE_B*Corr(s_A,s_B)).
    • Источник (paper): https://arxiv.org/abs/2411.00640 — 'Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations' (Miller, 2024)
    • Каветы: Framework paper with worked benchmark examples; developed for question-level benchmarks (MMLU-style), not generation quality — but the estimators are generic. Our 'question' = chapter, 'cluster' = book.
    • VERDICT: CONFIRMED
  • [LB/M] Ignoring clustering understates uncertainty by large factors in real evals: in Miller's Table 4, clustered vs naive SE ratios are 3.05x on DROP (questions nested in passages), 1.88x on MGSM, 1.10x on RACE-H. For chapters nested in books, book-level clustering of SEs is therefore mandatory, not optional.
    • Источник (paper): https://arxiv.org/abs/2411.00640 — 'Adding Error Bars to Evals' (Miller, 2024), Table 4
    • Каветы: Cluster inflation is dataset-specific; the chapter-in-book ICC for translation cohesion scores is unknown and must be estimated from a pilot.
    • VERDICT: CONFIRMED
  • [LB/M] Miller's power formula for a paired two-model comparison is n = (z_{alpha/2}+z_beta)^2 * (omega^2 + sigma_A^2/K_A + sigma_B^2/K_B) / delta^2, where omega^2 = Var(x_A)+Var(x_B)-2Cov(x_A,x_B) and K is resamples per question. Worked example: detecting a 3% absolute difference at 80% power / alpha=0.05 requires ~969 questions; with n=198 fixed, raising K from 1 to 10 shrinks the minimum detectable effect from 13.2% to 7.5%.
    • Источник (paper): https://arxiv.org/abs/2411.00640 — 'Adding Error Bars to Evals' (Miller, 2024), power analysis section
    • Каветы: The 969-question example assumes the paper's illustrative variance parameters; plug in pilot-estimated omega^2 for chapter-level cohesion scores. Numbers are per unclustered question — apply the design effect on top for book clustering.
    • VERDICT: CONFIRMED
  • [LB/M] Wang (FAIR, 'Measuring all the noises of LLM Evals') decomposes eval noise into prediction noise (sampling variability per question) and data noise (question sampling) via the law of total variance, measured over millions of question-level predictions. Paired prediction noise typically exceeds paired data noise, so averaging repeated generations per question detects effect sizes ~2x smaller for unrelated models and 6x+ smaller for related models (e.g., two variants of the same pipeline).
    • Источник (paper): https://arxiv.org/abs/2512.21326 — 'Measuring all the noises of LLM Evals' (Sida Wang, Dec 2025; interactive results at all-the-noises.github.io)
    • Каветы: Single author / single origin; measured on standard capability benchmarks (code, QA), not translation-quality judging. But arms sharing a base pipeline are exactly the 'related models' regime, so K>1 generations/judgments per chapter is a cheap power multiplier.
  • [LB/M] Resolution Diagnostics (Kotawala, May 2026) gives the paired sample-size formula N* = ((z_{1-alpha/2}+z_{1-beta}) * sigma_D / |delta|)^2 with sigma_D^2 = pAqA + pBqB - 2rhosqrt(pAqApBqB), a resolution ratio q = N/N (resolvable iff q>=1), and a clustering design effect DE = 1 + (m_bar - 1)ICC(D) that scales N. At alpha=0.05/power 0.8 with typical benchmark parameters (p0.65, rho0.3), resolving 1–2 percentage-point gaps requires N in the thousands to tens of thousands.
    • Источник (paper): https://arxiv.org/html/2605.30315v1 — 'Resolution Diagnostics for Paired LLM Evaluation' (arXiv 2605.30315)
    • Каветы: Formulas are for binary outcomes; for continuous 0-5 quality scores use sigma_D of paired score differences directly. Recent preprint, apparently single-author, not yet peer-reviewed. Clustering example uses MMLU-Pro subject categories (raised unresolved pairs from 4/9 to 6/9).
  • [ /M] The common shortcut of taking an unpaired Cohen-h sample size and post-multiplying by (1-rho) underestimates required N by roughly a factor of two in the close-comparison regime; three of five standard calculators (Cohen 1988 textbook, G*Power 3.1, R pwr) silently inherit this error. Do not size the paired chapter experiment with an unpaired calculator plus a correlation discount.
    • Источник (paper): https://arxiv.org/html/2605.30315v1 — 'Resolution Diagnostics for Paired LLM Evaluation' (arXiv 2605.30315)
    • Каветы: Affects the prereg's power calculation procedure, not the pipeline itself. Proven as a lemma for binary metrics near p=0.5.
  • [ /M] Card et al. (EMNLP 2020) meta-analysis found underpowered experiments are common in NLP; for machine translation, a typical 2000-sentence test set has only ~75% power to detect a 1-BLEU-point difference. They release notebooks for NLP power analysis.
  • [ ] For multi-arm / multi-dataset comparisons, Dror et al. (TACL 2017) provide a replicability-analysis framework based on partial conjunction testing with Bonferroni and Fisher variants (for independent vs dependent datasets), shown to be more powerful than plain Bonferroni while keeping conclusions statistically valid — a citable multiple-comparison procedure for a prereg with several arms and two endpoints.
    • Источник (paper): https://arxiv.org/abs/1709.09500 — 'Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets' (Dror, Baumer, Bogomolov, Reichart, TACL 2017)
    • Каветы: Methodological paper; designed for multiple datasets rather than multiple arms — for a small number of pre-registered arm contrasts, Holm-Bonferroni on the primary endpoint family is the simpler defensible choice.
  • [LB/M] Zheng et al. (NeurIPS 2023, MT-Bench/Chatbot Arena) established that a strong LLM judge (GPT-4) reaches over 80% agreement with human preferences on pairwise chat comparisons — the same level as human-human agreement — while documenting position bias, verbosity bias, and self-enhancement bias as systematic judge failure modes that need explicit control.
    • Источник (paper): https://arxiv.org/abs/2306.05685 — 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena' (Zheng et al., NeurIPS 2023)
    • Каветы: Single origin paper that most later judge-agreement claims re-cite — treat the '80%' as one voice. Measured on short chat responses in 2023, NOT on document-level translation cohesion; agreement on long-form cohesion must be re-measured against the human reader in-experiment.
  • [LB/M] Wang et al. 2023 measured that position bias alone can flip pairwise verdicts wholesale: by merely reordering candidates, Vicuna-13B could be made to 'beat' ChatGPT on 66 of 80 queries. Their mitigation, Balanced Position Calibration, evaluates each pair in both orders and aggregates — i.e., a minimum of 2 judge calls per pair with order swap is the floor for any pairwise protocol.
    • Источник (paper): https://arxiv.org/abs/2305.17926 — 'Large Language Models are not Fair Evaluators' (Wang et al., 2023)
    • Каветы: Measured with 2023 judges (GPT-3.5/GPT-4 era) on short outputs; bias magnitude on 2026 judges over chapter-length texts may differ but the swap-test control is standard regardless.
  • [LB/M] A 2026 reliability study ('The Coin Flip Judge?', GPT-4o-mini and GPT-4.1-mini judges, 29 tasks) measured a mean pairwise verdict flip rate of 13.6% across repeated calls (28% of questions exceed 20% flip rate), dropping to 2.8–7.9% at temperature=0. It recommends >=10–11 repeated trials with majority voting and randomized position to recover the 50-trial reference verdict with 95% probability; high-variance items need 15+ trials.
    • Источник (paper): https://arxiv.org/abs/2606.13685 — 'The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation' (arXiv 2606.13685, 2026)
    • Каветы: Recent preprint, small judge models, general short-form tasks — the 10-trial number is a starting point, not gospel for chapter-length cohesion judging; per-pair trial count trades directly against the cost endpoint. Alternative supported by the noises paper: fewer trials per pair but more pairs.
  • [LB/M] WMT24 Discourse-Level Literary Translation task is the closest published protocol to chapter-level webnovel judging: test sets of 12 books / 239 chapters (Zh-En, 337K words) and 122 chapters (Zh-Ru); human evaluation by 3 professional evaluators on a 0–5 scale along two dimensions — General Quality (fluency/adequacy) and Discourse Awareness (consistency, word choice, anaphora) — reaching 0.86 Cohen's kappa on Zh-En; automatic metric was document-level sacreBLEU (d-BLEU) over concatenated chapters.
    • Источник (paper): https://arxiv.org/abs/2412.11732 — 'Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation' (also https://aclanthology.org/2024.wmt-1.58/)
    • Каветы: Human eval ran only on Zh-En (too few Zh-De/Zh-Ru submissions) — there is no published human chapter-level protocol validated for Zh-Ru specifically. Evaluators were professionals scoring absolute 0-5 per dimension, not pairwise preference.
  • [LB/M] The same WMT24 literary findings report a significant gap between human and automatic rankings for at least one system (NTU scored lowest in human eval, 2.58 general / 2.42 discourse, while looking acceptable on automatic metrics) — document-level BLEU-family metrics cannot be the sole cohesion endpoint.
    • Источник (paper): https://arxiv.org/abs/2412.11732 — 'Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation'
    • Каветы: One task edition, Zh-En only; still the direct precedent for keeping the human reader as primary judge of cohesion with LLM judges as secondary.
  • [ /M] ESA (Error Span Annotation, Kocmi et al. 2024) — continuous DA-style rating combined with error-severity span marking — delivers annotations 'faster and cheaper than MQM at the same quality level' without requiring MQM experts, and WMT24's general task used document-level human evaluation for official rankings; ESA is a citable middle-weight protocol for the human reader's chapter assessments.
    • Источник (paper): https://arxiv.org/abs/2406.11580 — 'Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation' (Kocmi et al., 2024); WMT24 metrics task context: https://www2.statmt.org/wmt24/metrics-task.html
    • Каветы: Validated on En-De WMT23 system outputs at segment/document granularity, not on literary chapter cohesion; adopt the 0-5 two-dimension literary protocol as primary and ESA-style span marking as optional enrichment.
  • [ /M] DITING/AgentEval (2025) is a published evaluation framework specifically for web-novel translation, scoring six dimensions including terminology localization, tense consistency, and zero-pronoun resolution via multi-agent LLM deliberation; it achieved the highest correlation with human judgments among seven tested automatic metrics, backed by 18K+ expert-annotated Zh-En pairs and a 300-pair meta-evaluation set (MetricAlign).
    • Источник (paper): https://arxiv.org/abs/2510.09116 — 'DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translation' (2025)
    • Каветы: Sentence-pair granularity, Zh-En only — evidence that LLM judging of webnovel-specific phenomena is viable, not a chapter-level cohesion protocol; exact correlation coefficients not extracted from the abstract.
  • [ ] van Miltenburg, van der Lee & Krahmer (NAACL 2021) is the standard citation for NLP pre-registration; its appendix provides question templates per study type including 'NLP engineering experiment' (hypotheses, data, evaluation metrics specified in advance), and argues for registered reports. A counterpoint discussion exists (Søgaard et al., EACL 2023).
    • Источник (paper): https://aclanthology.org/2021.naacl-main.51/ — 'Preregistering NLP research' (van Miltenburg et al., NAACL 2021); counterpoint: https://aclanthology.org/2023.eacl-main.6.pdf — 'A Two-Sided Discussion of Preregistration of NLP Research' (EACL 2023)
    • Каветы: Templates are generic per study type; no published OSF prereg example was found for a multi-arm LLM-pipeline experiment with cost and quality co-primary endpoints — ours would need to compose the NLP template with the OSF standard template.
  • [ ] The OSF preregistration template explicitly requires the elements the experiment needs to freeze: sample size and its rationale ('including all power calculations'), a stopping rule ('if... not full control over your exact sample size, specify how you will decide when to terminate'; sequential-analysis thresholds must be pre-specified), and a single pre-specified statistical method per hypothesis to preclude method-shopping.
  • [LB/M] Cost-of-Pass (arXiv 2504.13359, 2025) is the citable retry-adjusted cost framework: v(m,p) = C_m(p) / R_m(p), the expected cost per attempt divided by per-attempt success probability — under independent retries the expected number of attempts to first success is geometric, 1/R_m(p), so v is the expected monetary cost of one accepted output. Frontier cost-of-pass V_p = min over available models (optionally including a human-expert cost baseline); inference-time techniques (majority voting, self-refinement) enter through inflated C_m(p) and 'rarely justify the costs'.
    • Источник (paper): https://arxiv.org/abs/2504.13359 — 'Cost-of-Pass: An Economic Framework for Evaluating Language Models' (2025)
    • Каветы: Assumes binary pass/fail and independent identically-distributed retries; a translation pipeline with graded acceptance and partial retries (re-run only failed chunks/gates) needs a stated extension: cost-per-accepted-chapter = sum of all attempt costs (including judge/gate calls) divided by number of accepted chapters, with the acceptance criterion pre-registered.
  • [ /M] A 2025 LLM-as-judge deployment study operationalizes first-attempt reliability as ECR@1 (Evaluation Completion Rate), measuring 85.4%–100% across judge models 'with material cost implications via retries' — precedent for reporting per-arm retry rates alongside the cost endpoint rather than folding them silently into averages.
    • Источник (paper): https://arxiv.org/abs/2512.01232 — 'LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost' (2025)
    • Каветы: Domain is test-coverage judging, not translation; exact retry-to-cost formula not extractable from the abstract. Practitioner/industry blogs echo 'cost per successful task' but add no formal framework beyond Cost-of-Pass.

добор: g4-russian-target-norms

  • [LB] Russian fiction dialogue norm (Розенталь, Пунктуация, раздел 14, § 52): each reply given from a new paragraph is introduced by an em-dash (тире) WITHOUT quotation marks; quotes + dash separators are used only when replies run in-line without attribution; if author's words follow a reply, the тире before the next reply is dropped.
    • Источник (standard): http://old-rozental.ru/punctuatio.php?sid=160 — Розенталь, «Знаки препинания при диалоге» (§ 52)
    • Каветы: old-rozental.ru is an online reproduction of the handbook, not a scan of a specific edition; § numbering matches the widely used «Справочник по русскому языку. Пунктуация».
    • VERDICT: CONFIRMED
  • [LB] Розенталь § 47: in-line direct speech is enclosed in кавычки; direct speech starting a new paragraph is introduced by тире; internal (unspoken) thoughts are ALWAYS in кавычки regardless of position — so a sanitizer must distinguish speech vs thought before choosing dash vs quotes.
    • Источник (standard): http://old-rozental.ru/punctuatio.php?sid=155 — Розенталь § 47 «Оформление прямой речи»
    • Каветы: Online reproduction; thought-vs-speech distinction cannot be decided by regex alone — it needs upstream annotation from the translation stage.
    • VERDICT: CONFIRMED
  • [LB] Розенталь § 48: direct speech after author's words takes a colon and initial capital; the colon is licensed only when the attribution contains a speech/thought verb (сказал, спросил…) — otherwise no colon; ?/!/… go before closing quotes, the period after them.
    • Источник (standard): http://old-rozental.ru/punctuatio.php?sid=156 — Розенталь § 48 «Прямая речь после слов автора»
    • VERDICT: OVERSTATED — коррекция: Розенталь § 48: direct speech after author's words takes a colon and initial capital; ?/!/… go before closing quotes, the period after them. The colon is licensed not only by speech/thought verbs (сказал, спросил, подумал…) but also by nouns close in meaning (вопрос, ответ, слова, восклицание, крик, мысль) and by verbs of feeling/action that admit an added speech verb (обрадовался, удивился, улыбнулся, подбежал); the colon is omitted only when inserting «и сказал / и спросил / и воскликнул» is impossible or awkward.
  • [LB] Розенталь § 49: direct speech BEFORE author's words ends in comma (or ?/!/…) plus тире, never a period, and the attribution starts lowercase — a deterministic lint: «. —» followed by lowercase attribution verb is always wrong; «, —» with capitalized attribution is also wrong.
  • [LB] Розенталь § 50: author's words INSIDE a reply follow five closed patterns — «, — …, —» (lowercase continuation), «, — …. —» (capitalized), «? — …. —» / «! — …. —», «… — », and «: —» when the attribution has two speech verbs; quotes are never re-opened around the attribution.
    • Источник (standard): http://old-rozental.ru/punctuatio.php?sid=158 — Розенталь § 50 «Слова автора внутри прямой речи»
    • Каветы: The five sub-cases determine capitalization of the continuation — a checker must know whether the original sentence would have ended at the break point, which is not purely lexical.
  • [LB] Полный академический справочник под ред. Лопатина §§ 154–157: in «?!» the question mark always precedes; when ? or ! combines with ellipsis the mark stands in place of the FIRST dot (yielding exactly «?..» and «!..», never «?...»); a comma meeting an ellipsis is absorbed by it; the legal direct-speech sequences are «запятая, тире», «?/! , тире», «точка, тире», «многоточие, тире», «двоеточие, тире».
  • [ ] Publishing practice per Мильчин & Чельцова «Справочник издателя и автора» (3-е изд., 2009): first-level quotes in Cyrillic editions are ёлочки « », nested quotes are лапки „ “, third level single лапки — a normalization rule for quotes-within-quotes and for converting CJK 「」『』 or English “ ” quotes.
  • [LB] The Палладий (палладица) system is the generally accepted and de-facto only system for writing Chinese proper names in Russian, used across official documents AND художественные произведения; its purpose is unification, not phonetic fidelity, and full pinyin→palladica tables exist as machine-usable data.
  • [LB] The Поливанов system is the dominant standard for Japanese in Cyrillic; spellings like «ши/чи» arise from re-transcribing Hepburn romanization when names are taken from ENGLISH intermediate sources and are rejected by professional linguists — i.e., an en-pivot pipeline systematically leaks non-standard Japanese name forms into Russian.
    • Источник (practitioner): https://ru.wikipedia.org/wiki/Система_Поливанова — «Система Поливанова»
    • Каветы: Fandom enforcement is contested (си/ши holy wars documented, e.g., moluch.ru/archive/157/44284); some fan communities deliberately prefer Hepburn-derived forms, so the glossary should treat the transcription system as a per-book policy choice, deterministically enforced once chosen.
  • [ ] Peer-reviewed study of realia translation in xianxia webnovels into Russian (Маслахова & Баймуратова, 2022, «Современные исследования социальных проблем»; corpus: «Основатель Тёмного Пути» Мосян Тунсю and «Божественный доктор: дочь первой жены») found four strategies — calque/neologism, descriptive, approximate (genus-species), contextual — and estimated ~5–7 translator footnotes per chapter are needed for culture-bound terms.
  • [LB] Professional Russian translators of Chinese novels (interviews, Афиша Daily) name the multi-appellation naming system as a core difficulty — one character (e.g., Lan Wangji) carries about six designations (birth name, courtesy name, title, А-/-эр diminutives) — plus dense classical-literature allusions; so a book glossary must model alias clusters per character, not 1:1 name mappings.
  • [ ] tl.rulate.ru (the main Russian collective webnovel-translation platform) documents: team access-right tiers with four member statuses, translation moderators, per-fragment reader error reports with typed error categories routed to the project owner for acceptance, and a separate «AI-переводы» catalog category segregating machine translations.
    • Источник (official-doc): https://tl.rulate.ru/site/help — «Как пользоваться сайтом», Tl.Rulate.ru
    • Каветы: The public help page does not document project glossaries in detail, though search snippets indicate per-project glossaries editable by the team exist; could not verify structure without login.
  • [ ] The Rulate community itself teaches translators standard Розенталь dialogue formatting — new line plus тире per reply, period changed to comma before attribution, capitalization patterns after attribution breaks — plus a stated allowance to render 2–3 short exchanges in-line with кавычки; i.e., readers of translated webnovels expect canonical Russian dialogue typography, and community norms match the codified rules.
    • Источник (practitioner): https://tl.rulate.ru/users/191249/blog/852 — «Как писать диалоги», Rulate user blog
    • Каветы: Community blog, not platform policy; but it independently reproduces Розенталь §§ 47–52, confirming those rules are the live community norm, not just prescriptive theory.
  • [LB] wuxiaworld.ru's documented crowd-editing workflow targets, in this order: grammar/spelling (LanguageTool recommended), mass replacement of character names and foreign terms via comment requests, character GENDER/pronoun errors, and empty-paragraph cleanup — direct community evidence that gender-agreement drift and name inconsistency are the dominant visible defects of machine/amateur translation into Russian.
    • Источник (official-doc): https://wuxiaworld.ru/redaktirovat-ranobe-novelly-manga/ — «Как редактировать Новеллы и Ранобэ», WuxiaWorld.ru
    • Каветы: Fan-site help page, not a measured defect taxonomy; but it converges with the VseGPT vendor statement below from an independent origin.
  • [LB] VseGPT's Нейропереводчик tool (LLM translation of novels with glossary injection into the prompt) states as its motivating problems: MT confuses character gender («мужчина/женщина»), translates the same term differently across passages, and violates fandom conventions from previous translators — the exact failure modes a deterministic glossary + cohesion checker must catch.
    • Источник (official-doc): https://vsegpt.ru/ExtTools/Neurotranslator — «Нейропереводчик с Глоссарием для переводов новелл», VseGPT; peer service: https://mlate.ru/ — «MLate.ru — Умный машинный перевод новелл и ранобэ»
    • Каветы: Vendor documentation with marketing flavor; no numbers. mlate.ru could not be fetched (socket errors) — existence confirmed via search only. Gender-confusion claim independently corroborated by wuxiaworld.ru editing docs (separate origin).
  • [LB] Chinese lacks grammatical tense; aspectual meaning is carried by markers 了/着/过 which do not map one-to-one onto the Russian perfective/imperfective + 3-tense system (imperfective: past/present/future; perfective: past/future) — so narrative tense and aspect in the Russian target are translator-created artifacts requiring contextual inference, with no source-side anchor to check against.
    • Источник (paper): https://cyberleninka.ru/article/n/kategoriya-vida-v-grammatike-kitayskogo-i-russkogo-yazykov — Пэн Фань, «Категория вида в грамматике китайского и русского языков», Челябинский гуманитарий, 2017; tense system: https://journals.susu.ru/lcc/article/view/434/567 — Лян, «Способы выражения времени глагола в русском и китайском языках»
    • Каветы: Contrastive-grammar papers, not corpus measurements of translation errors; the design implication (tense flips at chunk seams) is an inference from the grammar contrast plus community defect evidence, not itself measured.
  • [ ] Paragraph division in Russian is a compositional-stylistic, author-controlled device distinct from semantic-syntactic segmentation — the same text may legitimately be paragraphed differently — which legitimizes re-paragraphing one-sentence-per-paragraph CJK webnovel prose during translation rather than mirroring source line breaks.
    • Источник (paper): https://evartist.narod.ru/text14/11.htm — Валгина Н. С., «Теория текста», гл. «Абзац как композиционно-стилистическая единица текста»
    • Каветы: Quote taken from search-result rendering of the textbook chapter; full page not independently re-fetched. This is general text theory — no source found that specifically documents HOW ru webnovel translators merge paragraphs (see gaps).
  • [ ] Russian fan translations of webnovels/ranobe are predominantly made from the English translation (анлейт), not the original zh/ja, and readers attribute terminology calques and quality loss to this English pivot plus undisclosed machine translation sold as paid chapters.