# The control plane as a systemd unit. One VM, systemd, binaries from CI — not Kubernetes: a tmctl # child runs for HOURS and holds an exclusive lock on the book's files on the local disk, so any # orchestrator that can move a pod mid-run is hostile to this workload (PLATFORM_DIRECTION §3). # # ⚠ A RUN IS NOT A CHILD OF THIS UNIT (D39.106, ratified). Each one is a transient unit in this # user's OWN systemd manager, in tm-runs.slice, because a translation has to outlive a deploy or a # crash of the control plane and a child cannot. What follows from that, and is easy to get wrong: # # * `loginctl enable-linger tmplatform` is REQUIRED, once, at install: without it the user manager # does not exist outside a login session and no run can be created at all; # * this unit has to reach its own user bus, which is why ProtectHome= is `tmpfs` with a bind # rather than `yes` (measured: ProtectHome=yes makes /run/user/ inaccessible and the bus # unreachable). DBUS_SESSION_BUS_ADDRESS alone is enough — XDG_RUNTIME_DIR is not needed; # * the limits at the bottom of this file bound THE CONTROL PLANE ONLY. A run is bounded by its # own cgroup in tm-runs.slice (TM_PLATFORM_RUN_MEMORY_MAX / _TASKS_MAX) — that is the answer to # PD-13, and it is measured rather than declared (STACK_DECISIONS §16). [Unit] Description=TextMachine control plane After=network-online.target postgresql.service Wants=network-online.target # ⚠ ADD, at install, an ordering on this service user's own systemd manager: # After=user@.service # BindPaths=/run/user/%U below needs that directory to exist at start, and it is logind that creates # it. Without the line the first boot is a race; Restart=on-failure heals it after RestartSec, so the # symptom is a service that comes up on its second try rather than one that stays down. The UID is # site-specific, which is why it is not written here. [Service] # Type=exec, not notify: the binary does not speak sd_notify, and claiming it does would make # systemd wait for a readiness signal that never comes. Type=exec ExecStart=/usr/local/bin/tmplatformd User=tmplatform Group=tmplatform # KillMode=mixed: SIGTERM to the main process only, so the platform runs its own drain and stops # its children the way the engine expects; SIGKILL to everything left when the timeout runs out. KillMode=mixed KillSignal=SIGTERM # This bounds the shutdown of THIS unit: the HTTP drain plus the queue stop, with slack. # ⚠ Two corrections, 04.09, and the second reverses the first attempt at fixing this comment. # (1) The edition before that justified the number by "the engine's stop grace (30s)". There is no # 30s grace anywhere: `stopGrace` is 10 MINUTES (internal/runner/runner.go:59) and it is stamped # as TimeoutStopSec on the TRANSIENT RUN unit (runner.go:166), not honoured by this one. # (2) The first correction then claimed no tmctl is a child of this unit — ALSO false, and in the # more dangerous direction. Only `translate` is a transient unit. Four engine invocations run as # DIRECT CHILDREN of this daemon: manifest/status (internal/runner/engine.go:201), the export # build (build.go:88), bank-apply (bankapply.go:73) and systemd-run itself (runner.go:107). # Those DO sit in this unit's cgroup and this timeout DOES apply to them. What the header's # D39.106 rules out is a translation being killed here — not every tmctl. TimeoutStopSec=90 Restart=on-failure RestartSec=5s # Secrets as credentials, not as environment: an environment variable is visible in # /proc//environ and is inherited by every tmctl child. The config reads *_FILE first. LoadCredential=dsn:/etc/tmplatform/dsn LoadCredential=oidc_client_secret:/etc/tmplatform/oidc_client_secret Environment=TM_PLATFORM_DSN_FILE=%d/dsn Environment=TM_PLATFORM_OIDC_CLIENT_SECRET_FILE=%d/oidc_client_secret # The service's own user manager, which is where run units are created. %U is this unit's UID. Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus EnvironmentFile=/etc/tmplatform/env # Books live outside the repository and outside /var/lib; the unit gets the one directory it may # write and nothing else. /srv/textmachine is the SERVER's books root — "~/books" is the owner's # decision for a development machine, and under this unit a home directory is not reachable at all # (see ProtectHome below). # # The directory must EXIST before the first start: with ProtectSystem=strict a ReadWritePaths= that # names a missing path fails the mount-namespace setup and the unit does not come up. The deploy # notes create it, and the "-" prefix that would make the line optional is deliberately not used — # a service with no writable directory should fail at start, not at the first write hours in. ReadWritePaths=/srv/textmachine StateDirectory=tmplatform # Sandboxing. Free, and it bounds what a compromised process reaches. ProtectSystem=strict # /home and /root stay invisible — the platform has no business in anyone's home, which is also why # the books root is under /srv. An operator who must keep a library under a home directory adds it to # BindPaths= below alongside the runtime directory. # # tmpfs plus a bind of the user runtime directory, NOT `yes`: /home and /root stay invisible, and # the one thing that has to remain reachable — this user's systemd bus, through which every run is # created — does. Measured, both ways: with ProtectHome=yes the directory is Permission denied and # the bus is gone; with these two lines the socket is there. # ⚠ The path must exist when this unit starts, which is what `enable-linger` guarantees. ProtectHome=tmpfs BindPaths=/run/user/%U PrivateTmp=yes PrivateDevices=yes NoNewPrivileges=yes ProtectKernelTunables=yes ProtectKernelModules=yes ProtectControlGroups=yes RestrictSUIDSGID=yes RestrictRealtime=yes LockPersonality=yes RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX # These bound the unit's CGROUP: the control plane PLUS every engine process the daemon spawns # in-process — manifest/status, the export build, bank-apply — but NOT translation runs, which are # transient units in the user manager (the header's D39.106). # ⚠ Both earlier editions of this sentence were wrong, in opposite directions, and the pair is worth # keeping: the first said "every tmctl the platform spawns lives in it… the control plane plus every # run in flight, together (PD-55)" — pre-D39.106, and contradicted the header of this same file; the # 04.09 correction then said the cgroup holds the control plane and nothing else — which erased the # four in-process children that really are in it. PD-136 recorded the first paragraph as DELETED; it # was rewritten, not deleted, and only on 04.09. Two consequences follow, and both are decided # here rather than discovered in production: # # 1. The ceiling is a machine backstop, not a service sizing. systemd.resource-control(5) calls # MemoryMax= "the last line of defense"; as a percentage it needs no knowledge of the box. # 2. OOMPolicy is set explicitly. The system default is `stop`, which would take the control plane # down over one OOM inside it and land the unit in oom-kill failed state for Restart= to pick up. # `continue` logs the kill and keeps the service running. # # ⚠ These numbers bound the CONTROL PLANE and nothing else. Runs are not in this cgroup at all — each # is a transient unit in tm-runs.slice with its own MemoryMax/MemorySwapMax/TasksMax # (TM_PLATFORM_RUN_*). That is the answer to PD-13, and it is measured (STACK_DECISIONS §16). MemoryMax=80% OOMPolicy=continue # ⚠ The previous justification counted "platform plus concurrent RUNS" — a surviving conclusion of # the premise removed above: runs are not in this cgroup. What to count is the daemon plus its # IN-PROCESS children (manifest/status, the export build, bank-apply, systemd-run), each a Go # process with a few dozen threads. The number is unchanged: it is generous for that set too. TasksMax=512 LimitNOFILE=8192 [Install] WantedBy=multi-user.target