This repository has been archived on 2026-09-17. You can view files and clone it, but you cannot make any changes to its state, such as pushing and creating new issues, pull requests or comments.
nixos-config/hosts/pinix/localServices/prosody/README.md
pijukka d07e7c3d90 feat(prosody): migrate XMPP stack from pinix to blunix
Moves the Prosody server, xmpp-web frontends (chat/plaza), and Forgejo's
wiki-update webhook notifier from pinix to blunix, co-located with
Traefik/ACME so TLS comes straight from Traefik's cert store instead of
the old manual copy process. pinix's prosody/xmpp-web/webhook modules
stay in place but deactivated as a fallback, matching the earlier Forgejo
migration's pattern. Implements docs/intent/xmpp-umzug-pinix-blunix.md.

Deployed and verified live on blunix (nixos-rebuild switch, chat/plaza
returning 200, prosody healthy, agent XMPP listeners reconnected). Two
first-bootstrap bugs found and fixed during that verification, both
specific to bringing Prosody up on a host it never ran on before:

- prosody-certs-sync.service used to call a blocking
  `systemctl reload-or-restart prosody.service` whenever the cert file
  differed from what's installed -- including on first activation, where
  prosody.service's own start job is still queued behind this unit
  (Requires+After), deadlocking forever. Now only reloads an
  already-running Prosody; ordering alone brings up a fresh one with the
  cert just installed.
- hosts/blunix/localServices/secrets.nix force-owned the global `bindPW`
  secret to root with the (previously correct) rationale that it belongs
  to a pinix-only service. Prosody now runs on blunix too and needs to
  read it directly as its own LDAP bind config include, so that override
  no longer applies to it.

pinix's own deploy (data migration, DNS/webhook cutover) is separate
follow-up work, not part of this commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANg2L2xUmNJ45t9mcv6HVT
2026-09-06 16:46:26 +02:00

11 KiB

Prosody

Moved to blunix on 2026-09-06 (docs/intent/xmpp-umzug-pinix-blunix.md); see hosts/blunix/localServices/prosody/README.md for the active setup. This module and its data stay in place as a deliberate, disabled fallback (systemd.services.prosody.enable = false; in setup-prosody.nix) — everything below describes the old, now-inactive pinix topology, including the manual TLS copy process this move specifically replaced with an automated one on blunix.

Prosody ran on pinix. Public XMPP traffic reached port 5222 on blunix, where Traefik forwarded it through the Headscale/Tailscale network to 100.64.0.3:5222.

The service uses the upstream NixOS Prosody module. LDAP authentication is enabled by adding lualdap to the Prosody package's Lua environment in setup-prosody.nix; no store paths or copied NixOS module are required.

The LDAP password and MOTD are included from agenix files at runtime. Consequently, the module's build-time configuration check is disabled and the live configuration must be checked after activation.

Web clients

xmpp-web.nix builds two frontends from the same pinned xmpp-web release:

  • chat.0ff.space is the registered-account client;
  • plaza.0ff.space is the anonymous guest lobby and redirects directly into lobby@muc.0ff.space.

Both builds receive the same generated 0ff-space.css. Its background comes from the shared hosts/globalServices/branding/0ff-space-background.css, also used by LLDAP and Phanpy. It reproduces the launcher's 15-pixel dotted pattern and follows prefers-color-scheme for its light and dark variants. The stylesheet is copied into each immutable build output and injected into its index.html; there is no separately managed file below /var/www.

The module also keeps the launcher's five-color palette in one launcherPalette attribute set and maps xmpp-web/Bulma's semantic colors to it: primary uses strawberry red, danger neon red, success pink, info violet, and warning orange. Base, hover, active, and focus colors are replaced during both frontend builds, so the upstream teal, blue, and green accents do not reappear during interaction.

nginx terminates TLS on the Tailscale address 100.64.0.3:8443, serves the two static frontends, and proxies /xmpp-websocket to Prosody on 127.0.0.1:5280.

On blunix, traefik-xmpp-web.nix uses TCP TLS passthrough for the public web chat hostnames. TLS is therefore terminated on pinix; blunix neither receives the certificate key nor sees XMPP message contents.

Guests authenticate anonymously against anon.0ff.space and join lobby@muc.0ff.space. The room is persistent and lukas@0ff.space is its owner, so the normal LDAP-backed account can join and administer it from Monal without a second account.

The MUC archive is retained for 30 days and cleaned every four hours. mod_muc_moderation is installed from the Prosody community-modules revision pinned by nixpkgs and enabled only on muc.0ff.space. Clients supporting XEP-0425 can therefore retract abusive messages from the room and its archive; the moderator must join with an owner or administrator account.

Update notifications

Forgejo sends push and pull-request events to the local webhook receiver on pinix. The receiver posts short notifications as forgeBot to update@muc.0ff.space using vikunja@bot.0ff.space. The repository hook targets http://127.0.0.1:9000/hooks/wiki-update; Forgejo permits that exact loopback address in addition to its default external targets.

The update room is members-only. The bot therefore needs a member affiliation in the room; its password remains in go-sendxmpp.age.

Second channel: agent@bot.0ff.space

Vikunja updates additionally go out as a direct (1:1) message from vikunja to agent@bot.0ff.space, alongside the existing vikunja@muc.0ff.space post. Unlike a MUC room, a 1:1 JID needs no room creation or affiliation dance — Prosody delivers chat messages between local accounts regardless of roster subscription. The point of the separate account is to have a single-JID message archive that only carries this one event stream, instead of parsing it back out of the shared MUC room log.

agent@bot.0ff.space is intentionally never added to a human XMPP client: go-sendxmpp sends plaintext (no OMEMO), so a client like Monal shows this conversation unencrypted next to otherwise-encrypted chats.

The listener now runs on blunix, where the persistent Codex checkout and CLI authentication already live. hosts/blunix/localServices/agent-codex.nix keeps go-sendxmpp --listen connected as agent, archives its stdout in the journal, and only queues messages from the explicit sender allowlist:

journalctl -u agent-xmpp-listener

Accepted messages are atomically written below /var/lib/agent-codex/inbox. Messages from the same sender that arrive within a three-second quiet window are combined into one queue job. Sending /neu is handled immediately after any already pending text and resets the sender's active conversation.

agent-codex-worker.path starts a separate oneshot worker, which invokes codex exec in /home/drone/nixos-config/agents/platform. The nested agents/platform/AGENTS.md supplies the XMPP role in addition to the root repository instructions.

The first turn uses JSONL output to capture Codex's thread.started ID. Successful replies store the sender-to-thread mapping below /var/lib/agent-codex/sessions; later turns use codex exec resume <SESSION_ID>. A mapping expires after 30 minutes without a successful turn. /neu removes it explicitly, so the next content message starts a fresh thread. Codex's own session rollout files remain in the isolated /var/lib/agent-codex/codex-home.

Every invocation ignores personal Codex configuration, uses the isolated CODEX_HOME containing only the copied authentication, explicitly selects a read-only sandbox, and sends only the final agent message back to the original XMPP sender. systemd serializes the queue; tmux is not involved. Failed jobs remain in /var/lib/agent-codex/failed for inspection.

The first successful reply to each allowlisted sender includes a one-time usage note. In normal use:

  • send related short fragments within three seconds to combine them;
  • wait for the reply before beginning a different request;
  • send /neu to discard the current conversational context;
  • wait 30 minutes to let the active mapping expire automatically.

The account password lives in agentXmpp.age, decrypted at runtime to /run/agenix/agentXmpp on blunix, owner drone. It is not decrypted on pinix after the listener migration. The listener reads it through a mode-0600 runtime configuration, so it is not exposed in process arguments.

Useful checks on blunix:

systemctl status agent-xmpp-listener agent-codex-worker.path
journalctl -u agent-xmpp-listener -u agent-codex-worker
find /var/lib/agent-codex -maxdepth 2 -type f

Run the unit tests from the repository root:

nix shell --inputs-from . nixpkgs#python3 --command \
  python3 -m unittest discover \
    -s hosts/blunix/localServices/agent-codex \
    -p 'test_*.py' -v

Enable and deploy

Enable the Prosody configuration in ../default.nix:

./prosody/setup-prosody.nix

Build locally on tnix, copy the closure to pinix, and activate it:

nrsp

Refresh the TLS certificate

Traefik renews the wildcard Let's Encrypt certificate on blunix, but Prosody reads separate certificate files on pinix. Copy the current certificate manually before the previous one expires. Let's Encrypt certificates are normally valid for about 90 days.

Run all commands from tnix.

1. Check source and installed certificates

Check the current Traefik certificate without exposing the private key:

ssh -T blunix \
  'sudo jq -r '"'"'.cloudflare.Certificates[] |
    select(.domain.main == "0ff.space") | .certificate'"'"' \
    /var/lib/traefik/acme.json |
    base64 -d |
    openssl x509 -noout -subject -issuer -dates -fingerprint -sha256'

Check the certificate currently installed on pinix:

ssh -T pinix \
  'sudo openssl x509 \
    -in /var/lib/prosody/certs/0ff.space.crt.pem \
    -noout -subject -issuer -dates -fingerprint -sha256'

2. Transfer certificate and key

Transfer both files directly over SSH. The private key is never written to the tnix filesystem or printed to the terminal.

set -o pipefail
ssh -T blunix \
  'sudo jq -r '"'"'.cloudflare.Certificates[] |
    select(.domain.main == "0ff.space") | .certificate'"'"' \
    /var/lib/traefik/acme.json | base64 -d' |
ssh -T pinix \
  'sudo sh -c '"'"'umask 022
    tee /var/lib/prosody/certs/0ff.space.crt.pem.new >/dev/null
    chown prosody:prosody /var/lib/prosody/certs/0ff.space.crt.pem.new'"'"''
set -o pipefail
ssh -T blunix \
  'sudo jq -r '"'"'.cloudflare.Certificates[] |
    select(.domain.main == "0ff.space") | .key'"'"' \
    /var/lib/traefik/acme.json | base64 -d' |
ssh -T pinix \
  'sudo sh -c '"'"'umask 077
    tee /var/lib/prosody/certs/0ff.space.key.pem.new >/dev/null
    chown prosody:prosody /var/lib/prosody/certs/0ff.space.key.pem.new'"'"''

3. Validate and install

Validate the certificate, private key, and file permissions on pinix:

ssh -T pinix 'sudo sh -c '"'"'
  set -eu
  cert=/var/lib/prosody/certs/0ff.space.crt.pem.new
  key=/var/lib/prosody/certs/0ff.space.key.pem.new

  openssl x509 -in "$cert" -noout -dates -fingerprint -sha256
  openssl pkey -in "$key" -check -noout

  cert_hash=$(openssl x509 -in "$cert" -pubkey -noout |
    openssl pkey -pubin -outform DER 2>/dev/null |
    sha256sum | cut -d" " -f1)
  key_hash=$(openssl pkey -in "$key" -pubout -outform DER 2>/dev/null |
    sha256sum | cut -d" " -f1)
  test "$cert_hash" = "$key_hash"

  stat -c "%U:%G %a %n" "$cert" "$key"
'"'"''

Expected permissions:

prosody:prosody 644 ...crt.pem.new
prosody:prosody 600 ...key.pem.new

Back up the existing files, install the validated files, and restart Prosody:

ssh -T pinix 'sudo sh -c '"'"'
  set -eu
  dir=/var/lib/prosody/certs
  stamp=$(date -u +%Y%m%dT%H%M%SZ)

  cp -a "$dir/0ff.space.crt.pem" \
    "$dir/0ff.space.crt.pem.previous-$stamp"
  cp -a "$dir/0ff.space.key.pem" \
    "$dir/0ff.space.key.pem.previous-$stamp"
  mv "$dir/0ff.space.crt.pem.new" "$dir/0ff.space.crt.pem"
  mv "$dir/0ff.space.key.pem.new" "$dir/0ff.space.key.pem"
  chmod 640 "$dir/0ff.space.key.pem"
  systemctl restart prosody
  systemctl restart nginx
'"'"''

The installed key is 0640 prosody:prosody: nginx is a supplementary member of the prosody group and receives read-only access. The temporary key remains 0600 while it is being transferred and validated.

4. Verify

Check the service and listeners:

ssh -T pinix \
  'systemctl is-active prosody
   ss -lnt | grep -E ":(5222|5269|5280|5281)[[:space:]]"'

Verify the certificate delivered through the public XMPP endpoint:

set -o pipefail
timeout 20 openssl s_client \
  -starttls xmpp \
  -connect 23.88.33.137:5222 \
  -xmpphost 0ff.space \
  -servername 0ff.space \
  </dev/null 2>/dev/null |
openssl x509 -noout -subject -issuer -dates -fingerprint -sha256