This repository has been archived on 2026-09-17. You can view files and clone it, but you cannot make any changes to its state, such as pushing and creating new issues, pull requests or comments.
nixos-config/docs/research/intent-board-state-workflow-ui-2026-09-11.md

233 lines
29 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Intent Board: code, workflow, and live UI audit
Date: 2026-09-11. Investigator: the primary Codex conversation, without delegated research. Scope: assessment only. No production cards, configuration, jobs, deployments, or implementation files were changed.
## Assessment
The board is a functioning orchestration application with significant implemented capability, but its end-to-end guarantees and user-facing state model lag behind that capability. It can dispatch work, split goals into dependency-aware children, review results, collect telemetry, and arrange deployments. It cannot yet consistently tell the operator what has actually succeeded, why work stopped, or which recovery action will work.
The most consequential findings are not cosmetic: review rejection can be parsed as approval; the deployment runner does not select the reviewed revision; and manual bookkeeping can become supposedly trusted deployment evidence. These behaviors are reproducible in isolated diagnostics. They do not establish that a wrong production deployment has occurred.
My recommendation is to stabilize outcome semantics, revision provenance, and recovery before adding further agents or workflow states. Preserve the existing intent/worker separation, isolated builder worktrees, deterministic operator validation, and report links.
## Evidence and method
- Read current board frontend, API, worker branches, completion watcher, orchestration, fanout, telemetry, tests, Nix wiring, and relevant builder/deployment code.
- Used the existing Graphify graph for navigation: BFS query `board intent phase review worker`, 101 matching/traversed nodes, output capped at 37 nodes. Outcome: useful navigation, not current-source proof. Graph dated September 7; no graph metadata updated.
- Inspected the live application in Chromium through Playwright at `http://127.0.0.1:8060/`, on blunix, around 18:45–18:49 CEST. Desktop 1440×1000 and mobile 390×844; light and dark appearances. Opened read-only intent and review dialogs, tested keyboard focus across polling, and simulated a failed API request inside that browser only. Browser routing blocked all methods other than GET/HEAD. No mutation request was attempted.
- The HTTPS front door `https://intent-board.0ff.space/` returned 401 without authentication. Browser inspection used the local backend; authenticated reverse-proxy/browser behavior was not tested.
- Served HTML and installed trigger, worker, completion watcher, fanout, orchestration, and deployment runner matched the examined repository files by SHA-256.
- Existing tests: 279 board/telemetry tests and 38 subtests passed; seven selected builder deployment/deferred-deployment tests passed. No system build or activation.
- Additional temporary diagnostics reproduced action/transition mismatches, review parsing, unpinned deployment command construction, interleaved state overwrite, incomplete downstream queue inspection, and a research-transition exception. Subprocess calls in deployment/review probes were mocked; no model was invoked.
- Historical audit `docs/research/intent-board-architektur-ux-evaluation.md` was context, not evidence of current behavior.
Temporary evidence: `/tmp/intent-board-browser-audit.json`, `/tmp/intent-board-audit-live.json`, `/tmp/intent-board-family-telemetry.json`, `/tmp/intent-board-audit-probes.py`, and screenshots `/tmp/intent-board-{desktop,dark,mobile,intent}.png`. Temporary files are not durable project artifacts.
## Actual architecture
Intent and research-intent Markdown files provide titles and source documents. Runtime JSON overlays phases, results, relationships, profiles, and actions. Manual ideas and planning drafts originate in runtime state. The API merges these sources; it does not reconstruct a complete event history from Git.
The frontend projects many backend phases into seven lifecycle groups. It renders server-provided actions but independently derives lifecycle grouping, family progress, and status labels. These separate derivations are a major source of inconsistency.
The trigger creates files in the board inbox. The board worker executes research/review/operator/planning/inspection work or forwards implementation work into the separate agent-codex builder queues. Two builder lanes run through separate instances and isolated Git worktrees. Completion records return to the board watcher. Positive review creates a persistent deployment request; a separate runner performs the switch and reports back. Telemetry is stored separately in SQLite.
This separation has useful boundaries: model output is validated before operator decisions are applied, ordinary builder jobs request deferred deployment, and the deployment runner fixes its allowed host and repository. However, a request containing a commit is not equivalent to a deployment built from that commit.
Sources: `board-agent/trigger.py:401`, `:1230`; `worker.py:935`, `:1089`, `:2001`; `completion_watcher.py:203`; `orchestration.py:585`; `agent-codex/deploy_runner.py:63`. Paths below are relative to `hosts/blunix/localServices/` unless otherwise stated.
## Live snapshot and representative cases
The API snapshot had 50 cards: 28 archived, six planned, five deployed, two in implementation, two failed inspectors, one inspector question, two negative reviews, two completed research cards, one waiting parent, and one blocked parent. These are backend phase counts, not visible top-level card counts.
1. **`full-system-audit-remediation`:** parent blocked, one of five children successful. `mariadb-bind-local` has a deployment completion; `codeberg-mirror-disable` and `nix-gc-daily` have negative reviews. The two remaining children are `no_changes`. Despite this, the parent displays “Kinder laufen” and an active wave, derived from children retaining implementation phases. The waiting/running presentation contradicts terminal outcomes.
2. **`lldap-fail2ban-protection`:** the builder report says it could not read the required logs or use web search and therefore did not implement a guessed filter. This is a missing-capability/evidence outcome. The card is in implementation with `no_changes`, offering only Archive.
3. **`gotosocial-orphaned-data-cleanup`:** the report challenges the task's source premise and contains a non-fast-forward push rejection. This should distinguish a task clarification from a publication failure. It is also `no_changes` with only Archive.
4. **`board-session-usage-indicator`:** waiting parent, zero of three children successful. One child needs input and two are planned/dependency-blocked. The parent appears in Running with green “Status: abgeschlossen,” describing the completed operator job rather than the unfinished goal.
5. **`board-intent-operator-orchestration`:** the inspector stored `repo_status=partial` and evidence, but the card exposes “inspected” and an old Builder report rather than a diagnosis report. Some stored evidence claims fields are absent that are present in current code. Its observation revision/freshness must be visible; this does not prove what was present when that older inspection ran.
6. **`researcher-live-eval-tooling`:** phase `deployed`, negative `review_decision`, and no deployment status. It still offers Repair. This is an observed contradictory record, not proof that the current runner deployed a rejected revision.
7. **`board-nested-child-intents` and `board-reviewer-visual-verification`:** failed inspection records remain visible with parse-related error details. Recent parser changes mean these historical messages are not evidence that the same output still fails today. Recovery still needs clear revalidation and a usable result view.
## State-by-state workflow and information needs
| Backend state/path | What happens | Current presentation / gap | Information and next action needed |
|---|---|---|---|
| `idea` | Manual runtime-only card | Backlog; no visible conversion into a planning draft | Refine into intent, edit idea, archive/delete |
| `entwurf_laeuft` | Planner intake runs | Running with conversation | Queued vs executing, last activity, runtime, failure recovery |
| `entwurf_frage` | Planner waits for answer | Needs attention; conversation and reply controls | Concrete question, retained answer, clear responsibility |
| `entwurf_fehlgeschlagen` | Intake failed | Needs attention; retry available | Failure category and whether retry can resolve it |
| Draft → `geplant` | Wrapper writes Markdown and replaces draft card | Ready; original transcript no longer exposed as a draft | Stable identity/link to the conversation that produced the intent |
| `geplant` | Intent available; no automatic ordinary build | Ready; Operator, two Builder choices, Diagnose | Acceptance criteria, dependency readiness, recommended primary action and why |
| `operator_angefordert` / `operator_laeuft` | Read-only operator returns direct/split/needs-human decision | Running | Queued/executing, decision and source revision |
| Operator direct | Scheduler queues implementation | Running | Chosen lane/profile, rationale, immutable attempt identity |
| Operator split → `wartet_auf_kinder` | Children dispatched as dependencies and capacity allow | Running; nested children and verbose badges | Distinguish runnable, queued, executing, blocked, and waiting-on-user counts |
| `durch_kinder_blockiert` | At least one child failed/rejected | Needs attention parent; can still say children running | Actual blocking cause per child and recovery that preserves the family |
| `operator_klaerung_noetig` | Operator needs help or failed | Needs attention | Required human decision, editable answer/replan path |
| `in_umsetzung` / `in_umsetzung_leicht` | Work handed to separate builder lane | Running or Needs attention depending on last status | Queue position, executing job, capability requirements, last activity |
| Builder `committed` → `review` | Completion queues review of recorded commit | Running | Review pending/running, exact source and result revision |
| Builder `limit_reached` | Automatic escalation while ladder remains; otherwise terminal | Escalation badges, then Needs attention | Attempts used, next profile, total budget, exhausted reason |
| Builder `no_changes` | Usually no phase change | Needs attention, often Archive only | Structured already-satisfied / needs-input / forbidden / publish-failed outcomes |
| `aufteilen_angefordert` → `aufgeteilt` | Legacy splitter writes children | Running → Done, even though children may remain unfinished | “Decomposed; child work outstanding,” not goal completion |
| Review → `planung_mit_feedback` | Negative result stored | Needs attention; often green “abgeschlossen” | Explicit rejected verdict, blocking findings, repair/replan choice |
| Review → `bereit_fuer_deployment` | Positive result immediately queues deployment | Ready to deploy | Whether deployment is automatic, queued, running, or awaiting a person |
| Deployment → `deployed` | Runner success accepted by watcher | Done | Actual activated revision/system generation and task-relevant health evidence |
| Deployment failure | Watcher routes to fixer failure | Needs attention | Deployment-specific error, retained artifact, retry-deploy vs repair distinction |
| `klaerung_reparatur` / `inspector_laeuft` | Typed inspection or repair request | Running | Clearly distinguish diagnosis from code mutation |
| Inspection completion | Result stored; original phase restored unless question | Often only “inspected” | Readable diagnosis, checked revision/time, evidence and required action |
| `inspector_rueckfrage` / `fixer_rueckfrage` | User answer persists; new session continues | Needs attention with answer box | Correct role label; avoid generic “answer the Fixer” for inspection |
| `fixer_laeuft` | Isolated implementation repair | Running | Attempt limit, exact target, changes, separate review required |
| Fixer `not_applicable` | Remains `klaerung_reparatur` | Falls into Running although terminal | Explain no applicable repair and offer routing/replanning |
| Fixer/inspector failure | Error retained | Needs attention; Archive can be offered but rejected | Action list must agree with transition validation |
| Research requested/running/finished/failed | Separate research lifecycle; no Builder trigger | Ready/Running/Done/Needs attention | Source intent, report, retry-research, use findings to create linked intent |
| `archiviert` | Bookkeeping, not cancellation | Archived; all archived cards expanded | Reactivate, archive provenance, cancellation semantics, retained family visibility |
Sources: `trigger.py:360`, `:925`, `:1399`, `:1507`; `worker.py:1851`, `:2001`; `completion_watcher.py:350`; `board.html:816`, `:924`, `:1210`, `:1302`.
## Priority findings
### P1 — Negative review text can release deployment
`worker.py:1450` searches the entire report for an approval phrase before searching for rejection. Both isolated inputs returned `bereit_fuer_deployment`:
- `Not ready for deployment. Decision: back to planning.`
- `Nicht bereit für Deployment. Entscheidung: zurück zur Planung.`
The positive branch of `process_job` immediately queues deployment (`worker.py:2177` vicinity). Negation and quoted prior decisions are enough to undermine the gate; no malicious input is required. Use one validated structured verdict bound to a full commit hash, reject conflicting/ambiguous results, and separate report prose from the decision. Add negative/quoted/multiple-verdict cases.
### P1 — The deployment request is pinned; the build is not
`agent-codex/deploy_runner.py:63` accepts a request with `commit_hash` but fetches/fast-forwards `dev` and runs `switch --flake path:<shared-repo>#blunix`. The hash does not select or verify the build source. `_write_completion` (`:161`) reports the requested hash afterward.
A mocked command-trace reproduction confirmed that the requested hash appeared in none of the executed commands. If another commit reaches dev before deployment, it can be included without the corresponding review having completed. The mutable checkout also weakens reproducibility. Fix by building an immutable reviewed revision or a separately reviewed integration revision and recording the actual activated closure/revision. Merely comparing request/completion metadata does not close this gap.
### P1 — Mutable display state can masquerade as deployment evidence
An isolated `planung_mit_feedback → deployed` transition succeeded without a commit or deploy job. `deployment_evidence()` then returned `status=deployed`, source “trusted board wrapper state.” `apply_routing_recommendation()` also assigns phases directly instead of using the transition dispatcher: recommending `in_umsetzung` sets that phase while queuing zero jobs.
These do not directly execute unauthorized system changes in the reproduction; they create false progress/evidence and bypass action semantics. Separate operational facts from manual classification. Derive deployed status from a verified deployment record, and apply routing through the same validated command path used for ordinary actions. Sources: `trigger.py:1453`, `:1491`, `:1760`.
### P1 — Shared state updates are not transactional across processes
`trigger.py` has a process-local threading lock. Worker state writers (`worker.py:514`, `:544`), fanout, failure handler, metrics, and operator scheduling maintain separate read/modify/write implementations. The operator lock is separate; it is not a lock shared by all writers. Multiple writers use the same `.tmp` filename.
An isolated legal interleaving—trigger reads state, worker records completion, trigger saves its older snapshot—lost the completion field. This establishes the vulnerability, not an observed production incident. Atomic rename alone does not make the whole transaction atomic. Use one cross-process transaction mechanism, ideally a shared SQLite state/event store or a single writer, with unique temporary files if JSON remains.
### P1/P2 — Recovery semantics are missing, not just buttons
`builder.py:815` classifies anything without a recognized success record or the string `NICHT committed` as `no_changes`. `apply_commit():930` reports push failures in different prose. The live GoToSocial case shows the resulting misclassification.
`available_actions()` omits Diagnose for implementation phases; `repair_eligible()` excludes `no_changes`. Consequently these cards can be stranded with only Archive. Failed inspector cards conversely advertise Archive, but `update_phase()` rejects it until a reset to planned; the UI offers no such reset. Archived cards have no ordinary Reactivate action. `route_next` is an implemented task type without a normal action that requests it.
Return structured execution and publication outcomes from the harness, then derive both actions and state from those outcomes. Distinguish task clarification, missing tools/evidence, genuine no-op, failed checks, failed publication, and deployment failure. An infrastructure error should not automatically ask a code-writing Fixer to change the project.
### P2 — Queue/replay guarantees are incomplete
- `builder_trigger_blocked()` (`trigger.py:1202`) examines board queues only. After handoff, a downstream builder-processing record is not seen; an isolated same-card/lane probe returned no block. The operator scheduler does inspect downstream queues, so the two dispatch paths disagree. This is a guard gap; downstream serialization may limit simultaneous execution but does not establish duplicate-request prevention.
- Completion records move to `consumed` before processing (`completion_watcher.py:62`, `:529`). A crash between those steps leaves a consumed-but-unapplied result; the shown main loop does not replay it. This crash window was inspected, not triggered in production.
- A research reset from failed to requested writes state and then raises `UnboundLocalError` because `job_id` is initialized later in the non-research branch (`trigger.py:1598` vicinity). Reproduced in temporary state.
- Queue IDs based on slug and integer seconds and updates without current-attempt checks deserve explicit duplicate/stale-completion tests.
### P2 — The UI confuses job completion, goal success, and deployment
`statusLabel()` maps `done` to completed regardless of a negative review. Family status treats an implementation phase as active even with terminal `no_changes`. Lifecycle grouping prioritizes some phase labels over contradictory failure data. Child cards appear only inside their parent, so a child needing attention can be hidden in the Running column.
Have the server emit a single typed display state and explanation: goal state, current job state, last outcome, blocking actor, next action. Preserve both family navigation and a flat “needs my attention” view. Count children separately from parent goals.
### P2 — Diagnosis and provenance exist but are not adequately displayed
The API exposes repository evidence, deployment status, timestamps, profiles, and relationships, but much is never rendered. Inspector result summaries are not exposed as a readable report. The telemetry endpoint provides up to 20 runs; the dialog renders only `current_pipeline`. A Builder report is hidden once a review report exists. Review files are keyed by slug and overwritten, limiting accessible per-attempt provenance. Draft completion preserves its conversation in state, but ordinary intent serialization does not surface it.
Provide an intent detail view with Overview, Attempts, Evidence, Conversation, and Changes. Bind each observation to its source revision and time. Keep current recommendation distinct from old failures and old review verdicts.
## Browser and interaction findings
The desktop board is legible at the individual-card level, and dark mode is implemented consistently through theme variables. Native dialogs, keyboard handlers on report links, escaped text rendering, orphan fallback, and preservation of clarification text are useful foundations. No uncaught page error appeared during successful loading and read-only dialog use.
The screen gives scarce space to current work:
- Eight tracks including Agents; content spans 2014 pixels. At 1440 pixels, Done and Archived are outside the initial viewport.
- Agents occupies the first 260 pixels; Backlog another 230 pixels despite being empty. On a 390-pixel phone, initial content is mostly settings and test controls; actual work starts offscreen at x=538.
- The archived column was 5113 pixels tall; Needs attention 4382; Running 3128. Families expand all children and long operator explanations. There is no search, filtering, collapse, or urgency sorting.
- Dense colored family cards contain repeated parent/child identifiers, raw phase keys, mixed German/English labels, and three or more competing action buttons. This makes a hierarchy that is technically present difficult to scan.
- Prominent install/uninstall smoke-test controls occupy the main header. Everyday priority and freshness information is absent there.
- Intent/review dialogs show raw Markdown. They are useful evidence viewers but do not emphasize decision, outstanding criteria, or next action. The tested intent title also duplicated the “Intent:” prefix.
- A focused card Intent link became `BODY` after the five-second rerender. The edit guard protects input/select/textarea focus, not button or report-link navigation.
- Blocking the next board GET in the browser left the prior board displayed without a stale/offline indicator or live alert. The catch logs only to console.
Suggested direction: move Agents configuration into a settings drawer; show attention/running/ready summaries first; use collapsible families and a searchable archive; provide a list view at narrow widths; preserve keyed DOM nodes and keyboard focus; show last successful refresh and reconnect state. These are product recommendations from observed behavior, not a formal accessibility conformance audit.
## Metrics, operations, and maintainability
The live remediation family telemetry had ten jobs and reported 1,632,046 total tokens, of which 1,435,501 were cached input tokens, and 459.007 seconds of summed job duration. These are reported counters, not audited invoices or end-to-end elapsed time. Its reported cost aggregate was $0.9061986; coverage of missing costs is not indicated, so it must not be presented as a verified full pipeline cost.
Negative reviews and `no_changes` jobs had `success=true` in telemetry while the family remained blocked. That can sensibly mean a model call completed, but it must not be used as goal-success rate. The current dialog does not show the stored dollar cost and does not provide an attempt-history selector. Additional useful measures are time waiting for people, queue delay, time to accepted outcome, rework rate, verified deployment rate, missing-measurement coverage, and cost per accepted outcome.
All six live runtime configurations selected Claude at observation time. The user reports exhausted Claude session credits; the board does not expose provider availability in this view. Starting work should explain unavailable capability/quota and preserve the immutable chosen profile. The Codex UI currently lists only Default and `gpt-5.6-luna`; allowed model/configuration values are duplicated across frontend, orchestration, and Nix defaults.
The implementation spans roughly 14,000 lines across the board's Python/HTML files, including about 5,300 lines of tests. The concern is duplicated state rules, not line count itself. Documentation still describes the original single-transition pilot, dragging, and an automatic correction loop even though the current worker stores negative review results and exposes manual Repair. Tests preserve several old behaviors, including no-change staying in its phase, and many frontend assertions inspect source strings rather than browser behavior.
The inspected Restic configuration backs up LDAP, Headscale, and Forgejo dump paths, not `/var/lib/board-agent`. This is a backup coverage question, not proof that no other backup exists. Runtime state, queue/acknowledgment records, and telemetry need a documented consistency-aware restore procedure. Loaders that silently substitute an empty board for malformed state should expose degraded state rather than make old intents look newly ready.
Sources: `board.html:1117`, `:1152`, `:1564`, `:1667`; `telemetry.py:315`; `completion_watcher.py:267`; `worker.py:101`; `restic.nix:7`; `trigger.py:229`; `README.md:1`.
## Recommended implementation order
1. Repair the release boundary: structured review verdict, immutable deployment source, actual deployed revision evidence, no deployment claims from bookkeeping.
2. Introduce structured builder/publication outcomes and one shared transition/action contract. Cover every terminal outcome with a valid continuation; fix research reset and action mismatches.
3. Make state transactions and completion acknowledgment/replay reliable; enforce attempt identity and shared queue occupancy across both dispatch paths.
4. Make the board truthful: separate goal/job/deployment state; show explicit blockers and next actor; put diagnosis and per-attempt evidence in one detail view.
5. Improve layout and interaction: collapse settings/families/archive, attention-first list, search, mobile presentation, focus preservation, stale-data indicator.
6. Add invariant-focused tests and operational metrics: rejected verdict never releases; activated revision equals reviewed revision; advertised action is valid; every terminal failure is recoverable; concurrent updates survive; replay is idempotent; blocked children surface in attention.
There is no need for a framework rewrite to begin this work. A shared state/transition layer and small browser regression suite would address more risk than adding new roles.
## Uncertainties and product decisions
No production phase transitions, real retries, cancellation, repairs, model calls, deployment, destructive actions, or restore exercises were performed. Historical rows may reflect older code or manual intervention. The board snapshot is a point-in-time sample, and the two security-sensitive code findings were not tied to an incident. Authentication was checked only through the unauthenticated HTTPS response and configuration. No broad dependency/security audit or formal accessibility certification was attempted.
The main product decision is whether positive review should automatically deploy or should wait for explicit release. Current code automatically queues deployment, while “Ready to deploy” suggests a waiting gate. Either policy can be coherent, but the interface, authorization model, and evidence must say the same thing. A second decision is whether the default view serves a person managing goals or a person debugging agent infrastructure; today it heavily favors the latter.
## Implementation follow-up — September 12 (not yet activated)
The user selected automatic deployment after approval. The working-tree changes
implement structured commit-bound verdicts, immutable deployment builds, exact
activated-closure comparison, a deployment retry without another model run,
and refusal of automatic rollback from the last verified deployment. Manual
result-state assignment can no longer bypass worker evidence.
Board state now uses a shared cross-process transaction boundary, a durable
queue outbox and claim markers. Completion processing records acknowledgments
after transactional application and checks pipeline/attempt identity. The
research-reset failure and archived-family scheduling were corrected. Failed
publication preserves unpublished work; its location is shown as evidence.
The UI separates goal state from job outcome, surfaces blocked children and
unverified legacy deployments, and adds search, archive/attention views,
collapsible families, separate settings, detailed evidence/history/conversation
views, focus preservation and stale-data warnings. These changes were exercised
in Chromium with isolated fixtures and the earlier read-only live snapshot.
Regression verification: 419 Python tests and 38 subtests passed. Both Nix
script bundles were built; flake evaluation was checked without building the
VM checks or activating a system. The repeatable `test_board_browser.cjs` test
checks desktop/mobile layout, filtering, dialogs, keyboard focus and API failure
handling. It never sends live mutation requests or invokes a model.
Limits remain explicit: activation and live post-deployment smoke testing are
pending; old cards are not bulk-certified or restarted. Historical missing
attempt data cannot be recreated. Quota availability is not probed, and changing
model choices does not validate provider access. Restic coverage added here is
for board metadata and the deployment ledger, not a coordinated queue/SQLite
restore. A linked research-to-implementation planner flow and exhaustive
accessibility/production crash testing remain follow-up work. The original audit
above describes the observed pre-change system, not a claim that every suggested
product extension has been completed.
Final validation caveat: a later all-host `nix flake check --no-build` rerun,
including a retry outside the sandbox, failed while evaluating `tnix` because
a referenced cached flake source was no longer valid. This is not a passing
final all-host check; the earlier successful evaluation and script-bundle builds
do not replace resolution of that environment/input issue before a full rollout.
The scoped final `blunix` evaluation subsequently passed with `eval-cache=false`,
producing `rbh29vyl8lwnj0dw3ssjx3scwna8klfz-nixos-system-blunix-26.11.20260910.8ce4ef6.drv`.
No system was activated. Unrelated `flake.lock` and journald edits must be
excluded from any board-only activation snapshot.