textmachine/platform/internal/backup/backup.go

// Package backup makes restore points of the two things this deployment cannot re-create: the paid
// translation work, and the money registry.
//
// ⚠ WHY IT EXISTS AND WHAT IT IS FOR. A book's translated text lives in the engine's SQLite on the
// host's disk; credit, holds and the ledger live in Postgres. Until this package there was no copy
// of either anywhere (unified backlog row 269) — one lost disk was every user's paid work AND the
// record of who had paid what, with no way to restore it and no way to prove it. That is the one
// failure of this deployment with no partial outcome, which is why the answer here is deliberately
// unclever: whole files, whole dumps, a manifest that says what each one is, and a restore an
// operator performs with `cp` and `pg_restore` rather than with a tool of ours.
//
// # The shape: a restore point is a DIRECTORY
//
//	<BackupDir>/20260905T120000Z/
//	    manifest.json        — version, when, what is inside, sha256 and size of every file
//	    postgres.dump        — pg_dump --format=custom of the whole database
//	    books/<book id>/…    — the book's directory, recursively, minus what copyBookFiles skips and
//	                           RECORDS as skipped, plus the engine's consistent snapshot of its
//	                           project database as `project.db`
//
// A directory and not an archive, for one reason that outweighs the tidiness of a tarball: a
// restore point must be usable by an operator with nothing but a shell on a bad day, and a
// half-written archive is unreadable where a half-written directory is merely incomplete. It is
// PUBLISHED by rename from `.partial-<stamp>`, so a restore point that exists is a restore point
// that finished.
//
// # Three decisions this package makes, and why
//
//  1. **The ENGINE makes the database copy, never this side.** D39.85 forbids the platform to open
//     the engine's SQLite, and that rule is also the correct engineering: a live SQLite copied
//     byte-wise while a run writes it is silently torn. `tmctl backup` integrity-checks and
//     `VACUUM INTO`s (runner/backup.go), so what lands here is a file nobody is writing.
//  2. **Postgres has no partial outcome and books do.** A restore point without the money registry
//     cannot answer "who paid what", so a failed dump DISCARDS the whole point. A book that could
//     not be copied leaves the point publishable and marked incomplete: nine books saved beat none,
//     and the manifest says which one is missing rather than implying it is there.
//  3. **Integrity is proven twice, in two different senses.** The engine proves the SOURCE is a
//     sound database (`PRAGMA integrity_check`). This package proves the COPY is the source, by
//     recording a sha256 it can re-check later without the original (Verify). What it does NOT
//     claim is that the copy is a sound database — that would need opening it, which this side may
//     not do.
//
// # What is deliberately NOT backed up
//
// Built exports (`TM_PLATFORM_EXPORTS_DIR`) are TTL-ephemeral by design and rebuildable from the
// book; run state (`TM_PLATFORM_STATE_DIR`) is exit markers whose whole life is one run.
//
// ⚠ AND THE ENGINE'S OWN `backups/` SUBDIRECTORY INSIDE A BOOK IS NOT SUPERSEDED BY THIS — an earlier
// draft of this comment said it was, and that was wrong in the direction that matters. Those are
// PRE-RUN rollback points, taken by the engine's own guard before each paid run
// (backend/cmd/tmctl/backup.go, preflightBackup); a restore point here is taken on a SCHEDULE and can
// therefore already contain whatever a bad run did. They answer different questions — "undo that run"
// versus "the disk is gone" — and only the second is this package's. They are left untouched, and
// they are not copied because copying every previous copy on every pass is what makes a backup
// directory grow quadratically.
//
// Each of these is a decision and not an omission — see the runbook section this package's operator
// documentation lives in (deploy/README.md, «Бэкап и восстановление»).
package backup

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"io/fs"
	"log/slog"
	"net/url"
	"os"
	"os/exec"
	"path/filepath"
	"sort"
	"strings"
	"time"

	"textmachine/platform/internal/books"
	"textmachine/platform/internal/pgstore"
	"textmachine/platform/internal/runner"
)

// ManifestVersion is the restore point's document version.
//
// It carries one for the same reason every document crossing the engine seam does (17-seam-inbound-law
// п.3): a reader of a restore point may be a build of this platform written long after the point was
// made, and "what shape is this" must be answerable without guessing. It is the platform's own
// document, so the version is the platform's own.
const ManifestVersion = "tm-platform-restore-point-v1"

// ManifestFile, DumpFile, BooksDir and ProjectDBName are the fixed names inside a restore point.
// Fixed, because the restore procedure in the runbook names them and an operator types them.
const (
	ManifestFile  = "manifest.json"
	DumpFile      = "postgres.dump"
	BooksDir      = "books"
	ProjectDBName = "project.db"
)

// partialPrefix marks a restore point that is still being written. A leading dot keeps it out of a
// plain `ls` and out of the listing this package itself does: a point is only a point once renamed.
const partialPrefix = ".partial-"

// stampLayout is the restore point's directory name: sortable, filesystem-safe, and the same shape
// the engine stamps its own restore points with (backend/cmd/tmctl/backup.go, backupStamp).
const stampLayout = "20060102T150405Z"

// Store is the storage this needs. An interface so the service's decisions are testable without a
// database — the same shape the export door's Store has, and for the same reason.
type Store interface {
	BooksForBackup(ctx context.Context) ([]pgstore.BookDirectory, error)
}

// Engine is the engine half: `tmctl backup` against a book.
type Engine interface {
	Backup(ctx context.Context, binary, workdir string) (runner.BackupOutcome, error)
}

// Config is what an operator chooses.
type Config struct {
	// Dir is where restore points live. EMPTY DISABLES BACKUPS ENTIRELY, and that is a loud
	// default rather than a quiet one: the daemon says so at boot, because a deployment that
	// believes it has backups and does not is the state this package exists to end.
	//
	// Absolute, and the runbook asks for a path on a DIFFERENT device from the books and the
	// database — this package cannot check that (a bind mount and a symlink both defeat any test
	// of it), so it is documented as the operator's part of the bargain and named in the boot log.
	Dir string
	// Every is how often a restore point is taken. The pass rides the existing sweep ticker and
	// decides for itself whether it is due, so this is a real interval and not a multiple of the
	// tick.
	Every time.Duration
	// Keep is how many restore points survive. Older ones are removed after a new one is published
	// — after, never before: a prune that ran first would, on a host that then failed to write the
	// new point, leave fewer copies than it started with.
	Keep int
	// PgDumpBin and PgRestoreBin are the standard PostgreSQL tools. Named rather than assumed on
	// PATH because a deployment that installs Postgres under a prefix (the development stand does)
	// has them nowhere the daemon's PATH looks.
	//
	// ⚠ Their MAJOR version must match the server's: pg_dump refuses a server newer than itself.
	PgDumpBin    string
	PgRestoreBin string
	// DSN is the database to dump. It carries a password, which is why it never reaches an argv —
	// see dumpArgs.
	DSN string
	// EngineBinary is the versioned path of tmctl, the same one the runner and the export door use.
	// Empty means this instance cannot ask for a consistent database copy, and then books are
	// skipped and said to be skipped: a restore point that quietly held no book text would be the
	// worst object this package could produce.
	EngineBinary string
}

// Manifest is what a restore point says about itself.
type Manifest struct {
	Version string    `json:"version"`
	Stamp   string    `json:"stamp"`
	TakenAt time.Time `json:"taken_at"`
	// Complete is false when any book could not be copied. The Postgres half has no such flag: a
	// point without it is never published.
	Complete bool        `json:"complete"`
	Postgres File        `json:"postgres"`
	Books    []BookEntry `json:"books"`
	// Notes are the decisions a restoring operator has to know and would otherwise have to infer,
	// carried INSIDE the point rather than only in a runbook that may not survive with it.
	Notes []string `json:"notes"`
}

// File is one file inside a restore point.
type File struct {
	// Name is relative to the restore point's own directory.
	Name   string `json:"name"`
	Bytes  int64  `json:"bytes"`
	SHA256 string `json:"sha256"`
}

// BookEntry is one book inside a restore point.
type BookEntry struct {
	BookID string `json:"book_id"`
	// Workdir is where the book lived on the host that made this point — the path a restore puts it
	// back at, unless the operator is restoring onto a different layout.
	Workdir string `json:"workdir"`
	// Files are everything copied out of the book's directory, the consistent project database
	// among them (ProjectDBName).
	Files []File `json:"files"`
	// Skipped says why this book carries no PROJECT DATABASE in the point — NOT that it carries
	// nothing. A book whose intake has not finished has its source, its `book.yaml` and its journal in
	// here and no database, because no database exists yet; `Files` is what it does carry. Empty when
	// the database is present.
	Skipped string `json:"skipped,omitempty"`
	// NotCopied names what inside the directory was deliberately left out, with the reason. It is
	// recorded rather than assumed known: a restore point that quietly omits something is the object
	// this whole package exists to not be, and a reader six months from now cannot re-derive the
	// rules from the files that are present.
	NotCopied []string `json:"not_copied,omitempty"`
}

// Result is what one Take did, for the caller's log and metrics.
type Result struct {
	Stamp    string
	Path     string
	Books    int
	Skipped  int
	Bytes    int64
	Complete bool
}

// Service takes restore points.
type Service struct {
	Cfg    Config
	Store  Store
	Engine Engine
	Log    *slog.Logger
	// Now is the clock, injectable so a test can place two restore points in one second apart.
	Now func() time.Time
}

func (s *Service) log() *slog.Logger {
	if s.Log == nil {
		return slog.Default()
	}
	return s.Log
}

func (s *Service) now() time.Time {
	if s.Now == nil {
		return time.Now()
	}
	return s.Now()
}

// Enabled reports whether this deployment takes backups at all.
func (s *Service) Enabled() bool { return s != nil && s.Cfg.Dir != "" }

// Sweep takes a restore point if one is due. It is the daemon's entry point and rides the ordinary
// sweep ticker: the interval is decided HERE, against the newest point on disk, so the schedule
// survives a restart with no state of its own.
func (s *Service) Sweep(ctx context.Context) error {
	if !s.Enabled() {
		return nil
	}
	points, err := List(s.Cfg.Dir)
	if err != nil {
		return err
	}
	// ⚠ DUE-NESS IS JUDGED ON THE NEWEST POINT OF ANY KIND, not on the newest COMPLETE one — unlike
	// Age. The difference is deliberate: a deployment with one permanently uncopyable book would
	// otherwise find no complete point, decide a point is due, and take a full copy of itself on
	// every tick forever. The gauge is what must be honest about incompleteness; the schedule must
	// only not run away.
	if len(points) > 0 {
		if newest, perr := time.Parse(stampLayout, points[len(points)-1]); perr == nil {
			if s.now().Sub(newest) < s.Cfg.Every {
				return nil
			}
		}
	}
	_, err = s.Take(ctx)
	return err
}

// Take makes one restore point.
func (s *Service) Take(ctx context.Context) (Result, error) {
	if !s.Enabled() {
		return Result{}, errors.New("backup: no backup directory is configured (TM_PLATFORM_BACKUP_DIR)")
	}
	stamp := s.now().UTC().Format(stampLayout)
	final := filepath.Join(s.Cfg.Dir, stamp)
	if _, err := os.Stat(final); err == nil {
		// One restore point per second is the resolution of the stamp, and two in the same second
		// would silently merge. Refused rather than overwritten: this directory is the only copy of
		// something.
		return Result{}, fmt.Errorf("backup: a restore point named %s already exists", stamp)
	}
	if err := os.MkdirAll(s.Cfg.Dir, 0o750); err != nil {
		return Result{}, fmt.Errorf("backup: create the backup directory: %w", err)
	}
	staging := filepath.Join(s.Cfg.Dir, partialPrefix+stamp)
	// ⚠ Mkdir AND NOT MkdirAll, AND THAT IS THE MUTUAL EXCLUSION — measured, not theorised. With
	// MkdirAll two passes in the same second (the daemon's sweep and an operator's `tmplatformctl
	// backup`, or two instances) both "created" the same staging directory and then wrote into it
	// together: the first to finish RENAMED it away, the second's copies fell into a path that no
	// longer existed, and — far worse — the winner published a point whose only book had been skipped
	// because the ENGINE refused to overwrite the restore point the loser had just made under the same
	// timestamp. A point marked `complete` holding zero books is the worst object this package can
	// produce, and this is one of the two changes that stop it.
	//
	// Mkdir is atomic, so exactly one caller wins the second and the other does no work at all.
	if err := os.Mkdir(staging, 0o750); err != nil {
		if errors.Is(err, fs.ErrExist) {
			return Result{}, fmt.Errorf("backup: a restore point for %s is already being written (another pass, or a leftover of a killed one)", stamp)
		}
		return Result{}, fmt.Errorf("backup: create staging directory: %w", err)
	}
	// Every path out of here that is not a published point removes the staging directory: a leftover
	// `.partial-` would otherwise accumulate a full copy of the deployment per failure.
	published := false
	defer func() {
		if !published {
			if err := os.RemoveAll(staging); err != nil {
				s.log().Error("a partial restore point could not be removed", "path", staging, "err", err)
			}
		}
	}()

	m := Manifest{Version: ManifestVersion, Stamp: stamp, TakenAt: s.now().UTC(), Complete: true,
		Notes: restoreNotes()}

	pg, err := s.dumpPostgres(ctx, staging)
	if err != nil {
		// The money registry has no partial outcome (unified backlog row 269): a restore point
		// without it could not answer who had paid what, so it is not published at all.
		return Result{}, err
	}
	m.Postgres = pg

	books, err := s.Store.BooksForBackup(ctx)
	if err != nil {
		return Result{}, fmt.Errorf("backup: list books: %w", err)
	}
	total := pg.Bytes
	copied, skipped := 0, 0
	for _, b := range books {
		// ⚠ THE DEADLINE ABORTS THE POINT; IT DOES NOT TRUNCATE IT. Without this check the pass would
		// grind on past its budget: every remaining engine call fails instantly against a cancelled
		// context, each becomes a "hole", and the point is PUBLISHED anyway — incomplete, but with a
		// fresh timestamp. Since the books are listed oldest-first, the ones that would be dropped are
		// the newest — the ones being paid for right now — and every generation would drop them again.
		// A point that was never taken is an honest state the age gauge reports; a point that silently
		// omits this week's work is not.
		if err := ctx.Err(); err != nil {
			return Result{}, fmt.Errorf("backup: the pass ran out of its budget after %d of %d books; no restore point was published: %w",
				copied+skipped, len(books), err)
		}
		entry, err := s.copyBook(ctx, staging, b)
		if err != nil {
			// One book's failure is not the deployment's: the point is published, marked incomplete,
			// and the manifest names the book and the reason.
			// ⚠ NO BOOK ID. The zone's norm forbids a user's or a book's id in a log line
			// (ENGINEERING_STANDARDS §«структурные логи»), and acceptance has caught this exact class
			// here before (internal/httpapi/stream.go, "NO BOOK ID"). Nothing is lost by obeying it:
			// the MANIFEST names the book and the reason, which is the durable record an operator
			// actually reads, and this line's job is only to make the failure visible while it happens.
			s.log().ErrorContext(ctx, "a book could not be copied into the restore point; the manifest names which one", "err", err)
			// ⚠ AND WHAT IT HAD ALREADY COPIED IS REMOVED, which is the half that matters. A book that
			// failed PART WAY leaves real files in the staging directory — very possibly a truncated
			// `project.db` — and the entry that replaces it says "skipped" and lists none of them. They
			// would then be published, unhashed and unmentioned, and the runbook's restore step is a
			// plain `cp -a books/<id>/. <workdir>/`: the operator would copy a torn database back over
			// a good one while the manifest told them this book was not in the point at all.
			//
			// A skipped book leaves NOTHING. The manifest then describes the directory exactly.
			if rmErr := os.RemoveAll(filepath.Join(staging, BooksDir, b.ID)); rmErr != nil {
				s.log().ErrorContext(ctx, "a partly copied book could not be removed from the restore point",
					"err", rmErr)
			}
			entry = BookEntry{BookID: b.ID, Workdir: b.Workdir, Skipped: err.Error()}
			m.Complete = false
		}
		if entry.Skipped != "" {
			skipped++
		} else {
			copied++
		}
		for _, f := range entry.Files {
			total += f.Bytes
		}
		m.Books = append(m.Books, entry)
	}

	if err := writeManifest(staging, m); err != nil {
		return Result{}, err
	}
	// The point's contents are on the disk before its name is — every file is Sync'd as it is written,
	// and so is the manifest; this flushes the staging directory's own entries, so the rename below
	// moves a directory that is whole.
	if err := syncDir(staging); err != nil {
		s.log().Warn("the restore point's directory could not be flushed before publishing", "err", err)
	}
	if err := os.Rename(staging, final); err != nil {
		return Result{}, fmt.Errorf("backup: publish restore point: %w", err)
	}
	published = true
	if err := syncDir(s.Cfg.Dir); err != nil {
		s.log().Warn("the backup directory could not be flushed after publishing", "err", err)
	}
	// Pruning is AFTER the publish and its failure is not the backup's: a restore point that exists
	// is worth more than a tidy directory, and the next pass tries again.
	if err := s.prune(); err != nil {
		s.log().Error("old restore points could not be pruned", "err", err)
	}
	res := Result{Stamp: stamp, Path: final, Books: copied, Skipped: skipped, Bytes: total, Complete: m.Complete}
	// Counts and paths, never a money figure and never a book title (ENGINEERING_STANDARDS
	// §Наблюдаемость).
	s.log().InfoContext(ctx, "a restore point was written", "stamp", stamp, "books", copied,
		"skipped", skipped, "bytes", total, "complete", m.Complete)
	return res, nil
}

// restoreNotes is what a restoring operator must know and cannot see from the files.
//
// It lives INSIDE the point because a restore happens on the worst day this deployment has, possibly
// from a copy that travelled without the repository the runbook is in.
func restoreNotes() []string {
	return []string{
		"Restore Postgres with: pg_restore --clean --if-exists --no-owner -d <dsn> " + DumpFile,
		"Restore a book by copying " + BooksDir + "/<book id>/ back to its workdir, then renaming " +
			ProjectDBName + " to the file that book's book.yaml names in `project_db` — absent that key, <book id>.db.",
		"A book directory with NO " + ProjectDBName + " is not an error: that book's intake had not finished, so no " +
			"project database existed yet. Restore its files as they are.",
		"The engine's own pre-run restore points (a book's backups/ subdirectory) are NOT in here and are NOT superseded by these: " +
			"they are a different guarantee (a rollback point taken BEFORE a paid run, on the same disk) and are left untouched.",
		"Built exports and run-state markers are NOT in here: both are rebuildable and short-lived by design.",
	}
}

// dumpPostgres writes the money registry.
func (s *Service) dumpPostgres(ctx context.Context, staging string) (File, error) {
	out := filepath.Join(staging, DumpFile)
	args, env, err := dumpArgs(s.Cfg.DSN, out)
	if err != nil {
		return File{}, err
	}
	cmd := exec.CommandContext(ctx, s.Cfg.PgDumpBin, args...)
	cmd.Env = env
	var stderr strings.Builder
	cmd.Stderr = &stderr
	if err := cmd.Run(); err != nil {
		return File{}, fmt.Errorf("backup: pg_dump: %w: %s", err, firstLine(stderr.String()))
	}
	// The archive's TABLE OF CONTENTS is parsed before the point is published, which catches the
	// failure that actually happens: a truncated or empty file, a tool that wrote nothing, a version
	// mismatch that produced a header and stopped.
	//
	// ⚠ WHAT IT DOES NOT PROVE, said plainly because an earlier edition of this comment claimed more:
	// `pg_restore --list` reads the TOC and not the data blocks, so a green answer here means "this is
	// a readable archive", never "this restores". Only a restore proves a restore, and that is why one
	// is EXECUTED in the battery (backup_live_test.go) rather than argued for here.
	if s.Cfg.PgRestoreBin != "" {
		check := exec.CommandContext(ctx, s.Cfg.PgRestoreBin, "--list", out)
		var checkErr strings.Builder
		check.Stderr = &checkErr
		check.Stdout = io.Discard
		if err := check.Run(); err != nil {
			return File{}, fmt.Errorf("backup: the dump this pass wrote cannot be read back by pg_restore --list: %w: %s",
				err, firstLine(checkErr.String()))
		}
	}
	return describe(staging, out)
}

// dumpArgs builds pg_dump's argv and environment from the DSN.
//
// ⚠ THE PASSWORD NEVER REACHES AN ARGV. A connection string on the command line is readable by
// every process on the host through /proc/<pid>/cmdline, and the DSN carries a password
// (config.Config.DSN). It travels in the child's environment instead, which is the channel
// PostgreSQL's own tools document for it (PGPASSWORD) and which is scoped to the child.
func dumpArgs(dsn, out string) (args []string, env []string, err error) {
	// ⚠ THE WHOLE CONNECTION STRING IS FORWARDED, MINUS THE PASSWORD — it is not rebuilt from parts.
	// An earlier version passed host/port/user/dbname as four flags and dropped everything else,
	// which silently included `sslmode`: a deployment whose DSN says `verify-full` would have had its
	// nightly dump go out under libpq's default `prefer`, i.e. TLS optional and the certificate never
	// checked. Nothing would have said so — pg_dump succeeds either way. The same loss applies to
	// `sslrootcert`, `connect_timeout`, `application_name` and anything else an operator put there.
	//
	// The password is the one part that may not travel: a connection string on the command line is
	// readable by every process on the host through /proc/<pid>/cmdline. It goes in the child's
	// environment instead, which is the channel PostgreSQL's own tools document for it.
	u, perr := url.Parse(dsn)
	if perr != nil || u.Scheme == "" {
		// A key/value DSN (`host=… port=…`) is legal for libpq and cannot be edited as a URL. Refused
		// rather than mangled: the alternative is a dump taken under settings nobody chose.
		return nil, nil, fmt.Errorf("backup: the configured DSN is not a URL this platform can hand to %s without rewriting it; use the postgres:// form", "pg_dump")
	}
	password, _ := u.User.Password()
	if u.User != nil {
		u.User = url.User(u.User.Username())
	}
	args = []string{
		"--format=custom",
		"--file=" + out,
		// No prompt, ever: this runs unattended, and a tool that stops for a password is a pass that
		// hangs until its budget kills it.
		"--no-password",
		"--dbname=" + u.String(),
	}
	env = append(os.Environ(), "PGPASSWORD="+password)
	return args, env, nil
}

// copyBook puts one book's directory into the restore point, with a consistent database.
func (s *Service) copyBook(ctx context.Context, staging string, b pgstore.BookDirectory) (BookEntry, error) {
	entry := BookEntry{BookID: b.ID, Workdir: b.Workdir}
	if b.Workdir == "" {
		entry.Skipped = "the book has no directory on this host"
		return entry, nil
	}
	if _, err := os.Stat(b.Workdir); errors.Is(err, fs.ErrNotExist) {
		// ⚠ A MISSING DIRECTORY IS TWO DIFFERENT FACTS, told apart by the book's own status.
		//
		// A REJECTED intake may take its directory with it, so a row outliving one is ordinary. Any
		// other status means the directory should be there and is not: a books volume that did not
		// mount, or a tree that moved while the absolute workdirs in Postgres stayed where they were.
		// Calling that an ordinary skip is how a deployment comes to publish points holding only the
		// ledger, all marked complete, while the age gauge stays green.
		if b.Status == statusRejected {
			entry.Skipped = "the book's intake was rejected and its directory is gone"
			return entry, nil
		}
		return entry, fmt.Errorf("the book's directory %s is not there, and its status (%s) says it should be", b.Workdir, b.Status)
	} else if err != nil {
		return entry, fmt.Errorf("stat the book's directory: %w", err)
	}

	dest := filepath.Join(staging, BooksDir, b.ID)
	if err := os.MkdirAll(dest, 0o750); err != nil {
		return entry, fmt.Errorf("create the book's directory in the restore point: %w", err)
	}

	// ⚠ THE DIRECTORY'S FILES ARE COPIED FIRST AND UNCONDITIONALLY, BEFORE ANY QUESTION ABOUT THE
	// DATABASE. An earlier version answered "this book has no project database yet" by returning
	// immediately, and so took NOTHING — not the uploaded source, not `book.yaml`, not the journal —
	// while the point still called itself complete.
	//
	// That was wrong on this package's own terms: the source is the one file in the directory nobody
	// can be asked to supply again, and a book can sit in `parsing` indefinitely rather than for a
	// moment — the intake's `not_configured` and `storage_unavailable` never become terminal
	// (internal/books/parse.go), so a host with a broken book template holds every upload in that
	// state. Those uploads are precisely the ones with no other copy anywhere.
	files, err := s.copyBookFiles(staging, b.Workdir, dest, &entry.NotCopied)
	if err != nil {
		return entry, err
	}
	entry.Files = files

	if s.Cfg.EngineBinary == "" {
		// NOT a skip: an instance with no engine cannot copy ANY book's database, so every book would
		// be "skipped" and the point would claim to be a backup of a library it never opened.
		return entry, errors.New("this instance has no engine binary, so no consistent database copy can be asked for")
	}
	res, err := s.Engine.Backup(ctx, s.Cfg.EngineBinary, b.Workdir)
	if err != nil {
		return entry, err
	}
	if !res.Exited || res.ExitCode != 0 {
		// ⚠ THE ENGINE'S REFUSAL IS TWO DIFFERENT FACTS AND THEY MUST NOT BE ONE SKIP.
		//
		// A book whose intake has not finished — or was rejected — has no project database at all, and
		// the engine says so rather than failing: an ordinary state that leaves the point COMPLETE,
		// because the book's FILES are already in it and there is no paid work to lose. Every other
		// refusal is a book that HAS paid work which this point does not contain, and calling that
		// "skipped" the same way is how a point comes to say `complete: true` over a hole.
		//
		// The discriminator is the PLATFORM's own fact — the book's intake status — and never the
		// engine's exit code or its prose: this side cannot tell "no database yet" from "a broken
		// deployment" by a number, and reading it out of a message would be the same unversioned
		// coupling PD-449 already carries once.
		if notYetCut(b.Status) || b.Status == statusRejected {
			entry.Skipped = fmt.Sprintf("no project database yet (status %s), but the book's own files ARE in this point: %s",
				b.Status, res.Stderr)
			return entry, nil
		}
		return entry, fmt.Errorf("the engine made no copy of a book that should have one (exit %d): %s",
			res.ExitCode, res.Stderr)
	}

	// The consistent snapshot, under the fixed name the manifest's notes tell an operator to rename
	// back. The engine's copy is then removed: it was made for this pass, and leaving it would grow
	// the book's own directory by a whole database every cycle.
	f, err := copyInto(staging, res.Path, filepath.Join(dest, ProjectDBName))
	if err != nil {
		return entry, err
	}
	entry.Files = append(entry.Files, f)
	if err := os.Remove(res.Path); err != nil {
		// ⛔ THE PATH IS STRUCK OUT, and this line is why the rule had to become a mechanism: the copy
		// lives INSIDE the book's directory, so naming it names the book (`<books dir>/<book id>`), on
		// a WARN that the zone's standard keeps identifiers out of. Found by the acceptance of 11.09
		// (F4) — in this platform's OWN sentence, not in an engine's. What stays is the operation and
		// the errno; who the book is comes from the manifest this pass is writing.
		s.log().Warn("the engine's copy could not be removed after it was taken into the restore point",
			"path", books.WithoutItsPath(b.Workdir, res.Path),
			"err", books.WithoutItsPath(b.Workdir, err.Error()))
	}
	return entry, nil
}

// copyBookFiles copies the rest of a book's directory, recursively.
//
// ⚠ RECURSIVELY, because a book's own data can legitimately live in a subdirectory: `langpack_extend`
// is a per-book overlay of the pair's canon and `glossary_seed` may sit under one, both resolved
// against `book.yaml` (backend/internal/config/book.go). An earlier version of this function skipped
// every subdirectory and still published the point as COMPLETE — a book's canon lost in silence.
//
// What it skips, each for its own reason:
//
//   - **The live database and SQLite's sidecars of it** (`.db`, `-wal`, `-shm`, `-journal`). The
//     engine's consistent snapshot is already in as `project.db`; the live file is the torn one.
//     ⚠ Recognised by SUFFIX and not by name, deliberately the weak direction: the platform must not
//     derive the engine's project-database path (17-seam-inbound-law п.1), so this is a rule about
//     what NOT to copy. Its failure mode is copying a file it did not have to — a book whose
//     `project_db` is spelled without that suffix gets its live file copied BESIDE the snapshot,
//     which is untidy and never lossy (register row PD-453).
//   - **The engine's own `backups/`** — its pre-run restore points, which this point neither
//     supersedes nor replaces (see the package comment); copying every previous copy on every pass
//     is what makes a backup directory grow quadratically.
//   - **Dotfiles.** ⚠ THIS ONE IS A SECRET, NOT A TIDINESS RULE: the engine's convention is a `.env`
//     beside `book.yaml` carrying PROVIDER API KEYS (backend/cmd/tmctl/main.go, `loadDotEnv`), and
//     the runbook tells the operator to ship this directory OFF THE HOST. A backup that carried the
//     deployment's keys to wherever backups are shipped is a credential leak with a scheduler. Every
//     skipped name is recorded in the manifest, so nothing goes missing quietly.
//   - **Anything that is not a regular file** — following a symlink would copy from outside the book.
func (s *Service) copyBookFiles(staging, workdir, dest string, skipped *[]string) ([]File, error) {
	entries, err := os.ReadDir(workdir)
	if err != nil {
		return nil, fmt.Errorf("read the book's directory: %w", err)
	}
	var out []File
	for _, e := range entries {
		name := e.Name()
		switch {
		case strings.HasPrefix(name, "."):
			*skipped = append(*skipped, name+" (dotfile: may be the engine's .env of provider keys)")
			continue
		case e.IsDir():
			if name == engineBackupsDir {
				*skipped = append(*skipped, name+"/ (the engine's own pre-run restore points)")
				continue
			}
			sub, err := s.copyBookFiles(staging, filepath.Join(workdir, name), filepath.Join(dest, name), skipped)
			if err != nil {
				return nil, err
			}
			out = append(out, sub...)
			continue
		case isLiveDatabase(name) && !isTheUploadedSource(name):
			*skipped = append(*skipped, name+" (the live database or a SQLite sidecar of it)")
			continue
		case !e.Type().IsRegular():
			*skipped = append(*skipped, name+" (not a regular file)")
			s.log().Warn("a book's directory holds something that is not a regular file; it is not backed up",
				"name", name)
			continue
		}
		f, err := copyInto(staging, filepath.Join(workdir, name), filepath.Join(dest, name))
		if err != nil {
			return nil, err
		}
		out = append(out, f)
	}
	return out, nil
}

// engineBackupsDir is the engine's own restore-point directory inside a book
// (backend/cmd/tmctl/backup.go, backupDirFor).
const engineBackupsDir = "backups"

// notYetCut reports whether a book's intake has not reached the point where the engine creates its
// project database.
//
// `uploading` is a file still arriving and `parsing` is one the engine has not been asked about yet;
// everything after that has been through `tmctl manifest`, which creates and migrates the database
// (backend/cmd/tmctl/main.go, manifestCmd: "it opens the store exactly as `status` does — EXCEPT on
// the first touch of a project, where that path falls back to a full Open and therefore CREATES and
// migrates the database"). A `rejected` book is not listed here because its directory goes with its
// source, and the copier answers that case earlier and by looking.
func notYetCut(status string) bool { return status == "uploading" || status == "parsing" }

// statusRejected is the one intake outcome that legitimately leaves a book row with no directory
// (internal/books, the reject path removes the directory with the source).
const statusRejected = "rejected"

// liveDatabaseSuffixes are the live SQLite file and the two SQLite writes beside it. `.db-journal`
// is the rollback-journal name a non-WAL database uses.
var liveDatabaseSuffixes = []string{".db", ".db-wal", ".db-shm", ".db-journal"}

// isTheUploadedSource reports the one file the suffix rule above must never catch.
//
// ⚠ THIS IS NOT BELT-AND-BRACES, IT IS A MEASURED HOLE. The intake writes the upload as
// `source.<the user's own extension>` and keeps that extension verbatim — `extensionOf` is explicitly
// NOT an allowlist, because which formats exist is the engine's question (internal/books/books.go).
// So a reader who uploads `novel.db` gets `<workdir>/source.db`, and a rule that skips everything
// ending in `.db` would drop THE BOOK'S OWN TEXT from every restore point — silently, and in exactly
// the direction the rule's comment claims is impossible ("its failure mode is copying a file it did
// not have to"). The source is the one thing in that directory the user cannot be asked to supply
// again.
//
// It is recognised by the name the INTAKE gives it, which is this platform's own fact, not the
// engine's convention — so nothing here derives a path across the seam.
func isTheUploadedSource(name string) bool {
	return strings.HasPrefix(name, books.SourceName+".")
}

func isLiveDatabase(name string) bool {
	for _, s := range liveDatabaseSuffixes {
		if strings.HasSuffix(name, s) {
			return true
		}
	}
	return false
}

// copyInto copies one file and describes it. The copy is read back through the hash as it is
// written, so the digest in the manifest is of the bytes that landed rather than of the bytes that
// were read.
func copyInto(root, src, dst string) (File, error) {
	in, err := os.Open(src)
	if err != nil {
		return File{}, fmt.Errorf("open %s: %w", filepath.Base(src), err)
	}
	defer in.Close()
	if err := os.MkdirAll(filepath.Dir(dst), 0o750); err != nil {
		return File{}, fmt.Errorf("create %s: %w", filepath.Dir(dst), err)
	}
	out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o640)
	if err != nil {
		return File{}, fmt.Errorf("create %s: %w", filepath.Base(dst), err)
	}
	defer out.Close()
	if _, err := io.Copy(out, in); err != nil {
		return File{}, fmt.Errorf("copy %s: %w", filepath.Base(src), err)
	}
	// The data is on the disk before the manifest claims it is. Without this a host that lost power
	// between the rename and the flush would publish a restore point of empty files.
	if err := out.Sync(); err != nil {
		return File{}, fmt.Errorf("flush %s: %w", filepath.Base(dst), err)
	}
	if err := out.Close(); err != nil {
		return File{}, fmt.Errorf("close %s: %w", filepath.Base(dst), err)
	}
	return describe(root, dst)
}

// describe measures a file inside a restore point.
func describe(root, path string) (File, error) {
	rel, err := filepath.Rel(root, path)
	if err != nil {
		return File{}, fmt.Errorf("backup: name %s inside the restore point: %w", path, err)
	}
	sum, size, err := hashFile(path)
	if err != nil {
		return File{}, err
	}
	return File{Name: filepath.ToSlash(rel), Bytes: size, SHA256: sum}, nil
}

func hashFile(path string) (string, int64, error) {
	f, err := os.Open(path)
	if err != nil {
		return "", 0, fmt.Errorf("backup: open %s: %w", path, err)
	}
	defer f.Close()
	h := sha256.New()
	n, err := io.Copy(h, f)
	if err != nil {
		return "", 0, fmt.Errorf("backup: read %s: %w", path, err)
	}
	return hex.EncodeToString(h.Sum(nil)), n, nil
}

func writeManifest(dir string, m Manifest) error {
	raw, err := json.MarshalIndent(m, "", "  ")
	if err != nil {
		return fmt.Errorf("backup: render the manifest: %w", err)
	}
	path := filepath.Join(dir, ManifestFile)
	f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o640)
	if err != nil {
		return fmt.Errorf("backup: write the manifest: %w", err)
	}
	defer f.Close()
	if _, err := f.Write(append(raw, '\n')); err != nil {
		return fmt.Errorf("backup: write the manifest: %w", err)
	}
	// Synced like every file it describes. Without it a host that lost power just after the publish
	// would leave a directory that List reads as a restore point and whose manifest is empty — the
	// schedule would call that a point and the next one would be an interval away.
	if err := f.Sync(); err != nil {
		return fmt.Errorf("backup: flush the manifest: %w", err)
	}
	if err := f.Close(); err != nil {
		return fmt.Errorf("backup: write the manifest: %w", err)
	}
	return nil
}

// syncDir flushes a directory's own entries, which is what makes a rename survive a power loss —
// fsync on a file says nothing about the directory that names it. The engine does the same where it
// publishes by rename (backend/internal/pipeline/bankdecisions.go, syncDir).
//
// Best effort by design: a filesystem that refuses to open a directory this way must not turn a
// written restore point into a failed one.
func syncDir(path string) error {
	d, err := os.Open(path)
	if err != nil {
		return err
	}
	defer d.Close()
	return d.Sync()
}

// ReadManifest reads a restore point's manifest.
func ReadManifest(point string) (Manifest, error) {
	raw, err := os.ReadFile(filepath.Join(point, ManifestFile))
	if err != nil {
		return Manifest{}, fmt.Errorf("backup: read the manifest of %s: %w", point, err)
	}
	var m Manifest
	if err := json.Unmarshal(raw, &m); err != nil {
		return Manifest{}, fmt.Errorf("backup: the manifest of %s does not parse: %w", point, err)
	}
	if m.Version != ManifestVersion {
		// Loud, and this is the reason the document carries a version at all: a shape this build does
		// not know is not something to read half of.
		return Manifest{}, fmt.Errorf("backup: the restore point %s is version %q and this build reads %q",
			point, m.Version, ManifestVersion)
	}
	return m, nil
}

// List returns the stamps of the published restore points in dir, oldest first.
func List(dir string) ([]string, error) {
	entries, err := os.ReadDir(dir)
	if errors.Is(err, fs.ErrNotExist) {
		return nil, nil
	}
	if err != nil {
		return nil, fmt.Errorf("backup: read the backup directory: %w", err)
	}
	var out []string
	for _, e := range entries {
		if !e.IsDir() || strings.HasPrefix(e.Name(), ".") {
			continue
		}
		if _, err := time.Parse(stampLayout, e.Name()); err != nil {
			continue
		}
		out = append(out, e.Name())
	}
	sort.Strings(out) // the stamp is lexicographically ordered by construction
	return out, nil
}

// Verify re-reads a restore point and checks every file against the digest its manifest recorded.
//
// This is what makes the manifest more than a description: a copy that rotted on the disk, or one
// truncated by a full filesystem, is indistinguishable from a good one by size alone. It answers the
// list of what is wrong rather than the first thing, because an operator deciding which restore
// point to use needs to compare them.
func Verify(point string) ([]string, error) {
	m, err := ReadManifest(point)
	if err != nil {
		return nil, err
	}
	var problems []string
	check := func(f File) {
		sum, size, err := hashFile(filepath.Join(point, filepath.FromSlash(f.Name)))
		switch {
		case err != nil:
			problems = append(problems, fmt.Sprintf("%s: %v", f.Name, err))
		case size != f.Bytes:
			problems = append(problems, fmt.Sprintf("%s: %d bytes on disk, %d in the manifest", f.Name, size, f.Bytes))
		case sum != f.SHA256:
			problems = append(problems, fmt.Sprintf("%s: sha256 %s on disk, %s in the manifest", f.Name, sum, f.SHA256))
		}
	}
	check(m.Postgres)
	for _, b := range m.Books {
		for _, f := range b.Files {
			check(f)
		}
	}
	if !m.Complete {
		problems = append(problems, "the manifest says this restore point is INCOMPLETE: at least one book was not copied")
	}
	return problems, nil
}

// prune removes the oldest restore points beyond Keep, and any staging directory old enough to be a
// leftover rather than a pass in flight.
func (s *Service) prune() error {
	points, err := List(s.Cfg.Dir)
	if err != nil {
		return err
	}
	var errs []error
	if s.Cfg.Keep > 0 && len(points) > s.Cfg.Keep {
		// ⚠ THE NEWEST COMPLETE POINT IS NEVER EVICTED, however old it is, and that exception is the
		// whole difference between a retention policy and a countdown to data loss. Retention counts
		// DIRECTORIES; a deployment producing degraded points — an unmounted books volume, an engine
		// that stopped answering — produces them on the same schedule as good ones, so with the
		// defaults every point that actually holds the paid work is evicted in `Keep × Every` and the
		// operator's only signal is a gauge that went to +Inf some days after the last real backup
		// was already deleted.
		//
		// Keeping it costs one directory beyond the configured count, and only in the case where that
		// directory is the last copy of the deployment.
		keep := newestComplete(s.Cfg.Dir, points)
		for _, stamp := range points[:len(points)-s.Cfg.Keep] {
			if stamp == keep {
				s.log().Warn("the oldest restore point is kept beyond the retention count because it is the newest COMPLETE one; this deployment has been producing incomplete points",
					"stamp", stamp)
				continue
			}
			if err := os.RemoveAll(filepath.Join(s.Cfg.Dir, stamp)); err != nil {
				errs = append(errs, err)
			}
		}
	}
	// A `.partial-` older than a day is a pass that was killed — by a reboot, or by a deadline that
	// outran the defer. A day, so a pass genuinely still running on a slow host is never removed
	// under itself.
	entries, err := os.ReadDir(s.Cfg.Dir)
	if err != nil {
		return errors.Join(append(errs, err)...)
	}
	for _, e := range entries {
		if !e.IsDir() || !strings.HasPrefix(e.Name(), partialPrefix) {
			continue
		}
		stamp, perr := time.Parse(stampLayout, strings.TrimPrefix(e.Name(), partialPrefix))
		if perr != nil || s.now().Sub(stamp) < 24*time.Hour {
			continue
		}
		if err := os.RemoveAll(filepath.Join(s.Cfg.Dir, e.Name())); err != nil {
			errs = append(errs, err)
		}
	}
	return errors.Join(errs...)
}

// newestComplete answers the stamp of the newest point whose manifest says it is complete, or "" if
// there is none. It is what prune must not delete.
func newestComplete(dir string, points []string) string {
	for i := len(points) - 1; i >= 0; i-- {
		m, err := ReadManifest(filepath.Join(dir, points[i]))
		if err == nil && m.Complete {
			return points[i]
		}
	}
	return ""
}

// Age answers how old the newest COMPLETE restore point is, and whether there is one at all. It is
// what the daemon publishes as a gauge: "when did this deployment last have a copy of itself" is the
// only backup question worth alerting on.
//
// ⚠ COMPLETE, and the word is the whole value of the gauge. A point that could not copy a book is
// published — nine books saved beat none — but it is not a backup OF THAT BOOK, and taking its
// timestamp as the age would reset the one number an operator watches while the same book went
// uncopied in every generation. A deployment whose points are all incomplete reads `+Inf` here,
// which is exactly what it is: no complete copy exists.
//
// The manifests are read newest-first and the walk stops at the first complete one, so the ordinary
// case costs one file read.
func (s *Service) Age(now time.Time) (time.Duration, bool, error) {
	points, err := List(s.Cfg.Dir)
	if err != nil {
		return 0, false, err
	}
	for i := len(points) - 1; i >= 0; i-- {
		m, err := ReadManifest(filepath.Join(s.Cfg.Dir, points[i]))
		if err != nil || !m.Complete {
			// A point whose manifest cannot be read is not one this deployment can rely on either.
			continue
		}
		taken, err := time.Parse(stampLayout, points[i])
		if err != nil {
			continue
		}
		return now.Sub(taken), true, nil
	}
	return 0, false, nil
}

// firstLine keeps a child's diagnostic to one line for a log.
func firstLine(s string) string {
	if i := strings.IndexByte(s, '\n'); i >= 0 {
		s = s[:i]
	}
	return strings.TrimSpace(s)
}