21 lines
1.1 KiB
Text
21 lines
1.1 KiB
Text
# Universal sentence-terminator classes (pack-15): the ONE source of truth for the two consumers that
|
||
# must classify a terminator WITHOUT a book langpack — the source chunker (it segments a chapter into
|
||
# sentences) and the coverage gate (it counts sentences on both sides). Both used to carry their own
|
||
# copy of the same two switches, so a class edit had to be made twice or the two silently disagreed.
|
||
#
|
||
# A CONSTANT of the writing systems, not a per-book choice, so it is embedded with the binary and
|
||
# available to a no-langpack book (the ja→ru golden segments 。 here). The miner's pair-scoped
|
||
# terminator list is a DIFFERENT thing and stays in the pair langpack (configs/langpacks/<src>/
|
||
# sentence-terminator.txt): it answers "which marks end a sentence for THIS source language's alias
|
||
# co-occurrence", which a future pair may narrow, while these classes are the engine's segmentation
|
||
# alphabet that a book with no pack still needs.
|
||
#
|
||
# `category<TAB>rune`. cjk = splits ZERO-WIDTH (CJK prose has no inter-sentence space);
|
||
# generic = splits only before whitespace (ASCII/typographic).
|
||
cjk 。
|
||
cjk ！
|
||
cjk ？
|
||
generic .
|
||
generic !
|
||
generic ?
|
||
generic …
|