TriagoAI documentation

TriagoAI — Developer Guide

Everything a developer needs to run, build, test and navigate TriagoAI. For the

domain language see CONTEXT.md; for design/UI see docs/design.md; for known

issues and postmortems see docs/problem.md.

Stack

Commands

npm run dev          # next dev -p 3004   → http://localhost:3004
npm run build        # next build
npm run lint         # eslint
npm test             # vitest run
npx tsc --noEmit     # typecheck
npx prisma migrate dev      # create + apply migration locally (also regenerates client)
npx prisma generate         # regenerate client to lib/generated/prisma

Folder map

triagoai/
├── prisma/               # schema.prisma + migrations/ (SQL)
├── lib/
│   ├── agents/           # runtime.ts + 7 agents (triage/diagnosis/remediation/…)
│   ├── actions/          # "use server" actions (tickets, fleet, admin, auth, knowledge, …)
│   ├── fleet.ts          # simulated 10-node SME fleet + remediation executor
│   ├── live-generator.ts # continuous SME incident flow (Live Mode)
│   ├── llm.ts            # mock | kimi | deepseek | openrouter providers
│   ├── tools.ts          # allowlisted tool registry (read/write/destructive) + chat tools
│   ├── audit.ts          # SHA-256 tamper-evident audit chain
│   ├── rbac.ts           # ITIL role → permission matrix
│   ├── login-security.ts # brute-force protection + account lockout
│   ├── kb-library.ts     # sourced ITIL/ITSM article + report seed
│   ├── markdown.tsx      # minimal markdown renderer
│   ├── generated/prisma/ # generated Prisma client (do not edit)
│   └── billing.ts / plans.ts / sla.ts / tickets.ts / sso.ts / …
├── app/
│   ├── (app)/            # authenticated console (queues, tickets, dashboards, admin, …)
│   ├── (marketing)/      # public pages (home, pricing, security, about)
│   ├── api/              # route handlers (ai/chat, auth/login, stripe/webhook, …)
│   ├── evidence/[id]/    # print-optimized DORA evidence pack (outside app shell)
│   └── middleware.ts     # CSRF origin check on authenticated /api/* mutations
├── components/           # ui.tsx kit, sidebar, chat-client, demo-console, forms
├── docs/                 # developer.md, design.md, problem.md, adr/, roadmap-90d.md
├── security/             # policies, audit-log-schema, pen-test-plan, vendor-risk
├── tests/                # vitest suite (tests/**/*.test.ts + lib/**/*.test.ts)
└── scripts/              # send-daily-reports.mts, backfill-kb-library.mjs, fine-tune.mjs

Test conventions

serializes files to avoid SQLite lock flakes.

afterAll (tenant delete cascades to children).

lib/password.ts, lib/login-security.ts, lib/markdown.tsx).

ECC dev harness

The adopted ECC engineering system is wired locally and in CI:

typecheck · vitest · build · e2e · e2e:visual. One command that proves a change.

The Definition of Done and the evidence each stage produces live in

docs/agents/verification.md.

security-scan · test; --full adds agent-eval and build.

push runs npm run verify:ecc.

lib/ecc/rules.ts. See CONTEXT.md (ECC, Rule pack, Verify gate, Engineering

pipeline, Instinct) and docs/adr/0005, docs/adr/0006.

Gotchas (environment)

NODE_TLS_REJECT_UNAUTHORIZED=0 (for npm ci + Prisma engine downloads) — the

runner stage does NOT, so production outbound calls verify certificates.

sharp) can get truncated on npm install over the intercepted TLS — re-copy

from a known-good node_modules if a build suddenly fails.

port-binding gotchas. localhost:3004 is the container, not a local

next dev, unless you explicitly run npm run dev.

Schema changes (checklist)

the Dockerfile CMD).

Billing dunning

Tiered failed-payment recovery lives in lib/billing/dunning.ts (registered as

the hourly dunning job). On invoice.payment_failed the webhook classifies the

decline from charge.outcome.reason (fetched via the charge):

→ cancel immediately, no retries.

retry days, then cancel. ENTERPRISE [1,3,7,14], PROFESSIONAL [3,7,14],

STARTER [3,7]. Stop rules clear the sequence the moment the invoice is paid

or the subscription is cancelled.

Manual Stripe config (one-time): enable retention coupons in the Stripe

Customer Portal (Settings → Billing → Customer portal → Subscriptions and tick

"Allow customers to apply promotion codes") so a retention discount can be

offered before a customer cancels. Coupons themselves are created in the Stripe

Dashboard (or via API) — no code changes required.

RMM/PSA integrations

HaloPSA → Config → Integrations → Halo PSA API → Applications → New, auth

method Client ID and Secret, scope all, and set an "Agent to log in as"

(the app is scoped to that agent's team). Store per-tenant on the Tenant row

(haloEnabled, haloCompanyUrl, haloClientId, haloClientSecret). Smoke-test

with npm run verify:halopsa (HALO_CLIENT_ID/HALO_CLIENT_SECRET/

HALO_COMPANY_URL in env, or pass a tenant id). Ships disabled (gated on

haloEnabled).

ninjaOneEnabled; npm run verify:ninjaone.

Prospecting (Hunter.io)

lib/prospect-finder.ts finds/verifies MSP owner emails via the Hunter API

(gated on HUNTER_API_KEY, free tier 50 credits/mo). npm run enrich:prospects

parses local/gtm/targets-global.md, prioritizes the first ~25 domains, and

writes data/leads.csv (--limit=N, --domains=a.com,b.io for a smoke test).

Conventions

lib/audit.ts audit().

specs/<service>.md and run tests/spec-contract.test.ts (see AGENTS.md).

+ next build before pushing.

"Documentation map" section in CONTEXT.md.

TriagoAI — Deployment & Data Residency

TriagoAI is a single-container Next.js app backed by SQLite (libsql) with a

vectors.db RAG sidecar. This document covers the current single-region

deployment, the multi-region target, and self-hosting with local AI.

> See docs/adr/0001-sqlite-single-node.md for why SQLite single-node is

> deliberate for the showcase, and the trade-offs of the multi-region path.

1. Current — Hetzner (single region, eu-central)

The live reference instance runs on a Hetzner CX22 VPS in Falkenstein

(eu-central).

                        ┌───────────────────────────────┐
   Browser ──:80/3004──►│  Docker: triagoai (node:22)   │
                        │  next start -p 3000            │
                        │  ├─ dev.db      (SQLite)       │
                        │  ├─ vectors.db  (RAG sidecar)  │
                        │  └─ uploads/    (file library) │
                        └───────────────────────────────┘
                                   volume: opspilot-data → /app/data

tenant's Tenant.region field defaults to eu-central.

2. Multi-region — AWS / GCP / Azure

Multi-region is a deployment topology, not a code change. Each region runs the

same container; a global router pins a tenant to its home region.

                  ┌──────────────┐
   Browser ──────►│  Global DNS  │──(tenant→region map)──┐
                  │  / edge LB   │                        │
                  └──────────────┘                        ▼
                        │                    ┌───────────────────────────┐
                        │                    │ eu-central (ECS/EKS)      │
                        │                    │ triagoai container        │
                        │                    │ dev.db · vectors.db       │
                        └───────────────────►└───────────────────────────┘
                                             │ us-east (ECS/EKS)         │
                                             │ triagoai container        │
                                             │ dev.db · vectors.db       │
                                             └───────────────────────────┘

Cloud SQL if you migrate off SQLite).

Key points (conceptual, not a runbook):

lives; the router sends that tenant's traffic to its region. No tenant is

served from two regions at once (SQLite is single-writer — see ADR-0001).

for dev.db + vectors.db + uploads.

single-writer SQLite to Postgres/Turso; that is a Phase-G follow-up, not

this milestone.

exports are self-describing.

3. Self-host — Docker Compose + local AI

For customers who need everything in-box (regulated industries, air-gapped

sites), TriagoAI runs standalone with Ollama for AI — no data leaves the box.

# .env (self-host)
LLM_PROVIDER=          # unused when tenants are aiMode=local
OLLAMA_BASE_URL=http://ollama:11434
REGION=eu-central

then routes to the self-hosted Ollama instance instead of a cloud provider.

local completions).

4. Data residency

Two labels answer "where does my data live?":

In a multi-region deployment these stay in sync: a tenant provisioned in

us-east has region = "us-east" and is served by the us-east deployment.

The field exists now so the label is already present for audit and routing when

the multi-region topology is turned on.

TriagoAI — Design Guide

Product and visual design: the design system, the 7-agent/ITIL model, and the

industry-pack differences. Domain terminology lives in CONTEXT.md.

Design system — flat navy panels

A dark, flat navy UI driven by CSS custom properties in app/globals.css.

Shared primitives live in components/ui.tsx: Card, CardHeader, Badge,

Button (variants: default/ghost/outline/danger/success), Input, Textarea,

Select, Label, Table (Th/Td), Empty. Reuse these rather than ad-hoc

Tailwind; map status/priority through STATUS_COLORS/PRIORITY_COLORS in

lib/utils.ts.

The 7 agents (ITIL 4 mapping)

Every agent run records an inspectable timeline (think → tool → approval →

result). This transparency + the SHA-256 audit chain is the core differentiator

vs ServiceNow's "Now Assist".

Architecture

!TriagoAI Architecture

detect → triage → diagnose → remediate (human approval) → verify → document
                  └──────── tamper-evident audit chain (always on) ────────┘

global + caller-tenant only.

approveRemediation. Matrix at /roles, defined in lib/rbac.ts.

self-hosted Ollama path (per-tenant aiMode).

Industry packs

Key screens

See the ITIL-mapping table in README.md for the full practice → screen map.

---

Command centre — operator design (2026-09)

The internal command centre (command/dashboard.mjs, served at agents.triagoai.io

behind Basic Auth) is a separate application from the product UI. It was

rebuilt in Sept 2026 and its design is the reference for a calm dark-ops

surface.

Palette — dark-ops, not neon

The previous palette was neon (cyan/magenta/acid-green on near-black), which read

as a demo rather than a tool. The current one:

Rules: one accent, status colours carry meaning and are never decorative, flat

surfaces (no backdrop-filter, no per-panel glow), and dark ink on light paper for

*documents* (below).

Information architecture — one nav model, a rail, and a graph workspace

Nav model:   lib/nav-model.mjs — the single source the sidebar, the palette and
             the mobile drawer all read (Now · Agents · Growth · Ops)
Desktop:     icon rail by default, hover shows the label, a toggle expands the tree
Mobile:      the sidebar BECOMES the drawer (hamburger + scrim); no bottom tab bar
Top bar:     ☰ menu   ⇤ rail toggle   title   ✉ Chat   ⌕ Search   🔔 Notifications
Workspace:   /graph — a 2D force graph of the company (Director → departments →
             skills/SOPs, memory, knowledge hubs, milestones); /workspace and
             /mindmap 302 to it
Aliases:     /sops → /library · /report → /reports (302)
Everything:  reachable via the command palette (⌘K search, ⌘O switch)

The hardcoded four-tab bar is gone: it was a second source of truth that could drift

from the sidebar, and it hid seventeen tabs. Regrouping is safe; a tab is removed only

when the ticket that owns its replacement lands. assertNavModel() enforces that

every tab sits in exactly one group and every redirect points at a real tab.

(Adopted from our own agents/docs/obsidian-ux-research.md and graph-view-research.md:

palette, switcher, force-layout graph view, minimal chrome.)

Decision surfaces

daily plan awaiting approval, each with inline approve / request-changes / reject.

A decision surface, not a list.

rate, tool failures, and description proposals. It states plainly that *"done

without verification is work nobody checked"* and shows no flattering 100% when

there is nothing to measure.

Documents vs dashboards

Progress on a mission is a hill chart (Shape Up: uphill = being figured out,

downhill = being understood), not a percentage — a linear bar implies uniform

progress and hides the risky half. Mission reports are *documents*: light paper

(#fafaf9), slate ink (#0f172a), serif headings, emerald accent — deliberately

distinct from the dark operator chrome, because a document is read and a dashboard

is scanned.

Design rules learned here

reassuring idle dot. Every number comes from a table.

("the reconciler fills these", "the suite runs Mondays") rather than showing 0.

consumed, not just declared (ADR-0008).

tables, 40px touch targets, env(safe-area-inset-bottom), tables that scroll

instead of cramming.

TriagoAI — Third-Party / Vendor Risk Assessment

This is a design-time risk register for the external services TriagoAI can use.

Every integration is optional and off by default; the local-AI and mock

modes are the zero-sharing fallbacks.

Key controls

present. Absent keys → deterministic mock, so the demo runs with zero egress.

DeepSeek/OpenRouter/Kimi or a self-hosted Ollama endpoint; a tenant's

aiMode forces the local path so no prompts leave the customer's network.

ingested, and the audit/prompt boundary strips instructions from untrusted

input (UNTRUSTED_DATA_BOUNDARY).

environment-only and never written to logs, audit payloads, or client output.

Per-tenant bearer tokens for inbound webhooks

TriagoAI's inbound webhook intake (POST /api/webhooks/inbound) authenticates callers with a per-tenant bearer token: the tenant's webhookSecret field, generated on enable and rotatable from Admin → Webhooks. The token is sent as Authorization: Bearer <secret> and validated by direct DB lookup — no JWT, no shared infrastructure secret.

Alternatives considered:

Consequences:

Status: accepted.

TriagoAI Audit Log — Schema & Tamper-Evidence

This documents the append-only audit log that underpins the Financial pack's

compliance story (SOX / DORA alignment) and is the primary artefact for SOC 2

evidence collection.

1. Event types logged

Every mutating action writes an AuditEvent. Representative actions:

2. Schema fields

3. Tamper-evidence mechanism

Each event's hash is:

hash = SHA-256(prevHash + payload + action + entityId)

Because hash binds to the previous event's hash, the whole chain is

transitively linked: altering *any* field of *any* event changes its hash, which

changes the prevHash of the next event, and so on. Verification walks the

chain from genesis and compares each stored prevHash/hash against the

recomputed value — a single mismatch marks the chain broken at that event.

The export endpoint (/api/admin/audit/export) re-verifies the chain on every

export and stamps the document with a verified boolean and brokenAt id.

4. Retention & access controls

see the retention policy in security/policies.md).

members; verification and export are admin-only (can(user.role, "admin")).

the application; rows are only ever created via the audit() helper.

5. Export format

a header row and one row per event (all ten fields).

Both are produced by lib/audit-export.ts and served with

Content-Disposition: attachment.

TriagoAI Security Policy

This document records the security posture for the TriagoAI showcase. It is

written to be reviewed by a technical CTO or security assessor ahead of a SOC 2

Type I engagement. Where a control is marked demo, it reflects the current

single-node showcase deployment; production hardening is noted inline.

1. Access control

INCIDENT_MANAGER, PROBLEM_MANAGER, CHANGE_MANAGER, CAB,

SERVICE_OWNER, REQUESTER). Every server action is gated through

can(role, permission); there is no ambient trust.

audit, and destructive tools (restart, clear-disk, failover, rollback,

password reset) can never auto-execute — they park behind a human approval

gate gated on approveRemediation.

expiry, SameSite=Strict. Secure is opt-in (COOKIE_SECURE) so the demo works

over HTTP; production must enable it.

(POST /api/auth/login, POST /api/auth/signup) with persistent rate limits

and lockout: max 5 failed logins per IP per 15 minutes, max 10 failed logins

per account per hour, and a 15-minute account lockout after 5 consecutive

failures (tracked via User.failedLoginCount/User.lockedUntil). Every

attempt is recorded in LoginAttempt (email, IP, success, reason) — including

unknown-email probes. Exceeding a limit returns 429 with a retry hint.

(auth.login.failed, for known accounts) login is written to the

tamper-evident audit chain with the source IP and user-agent. Unknown-account

failures are recorded in LoginAttempt only (no tenant chain to attach to).

Defense-in-depth: a middleware.ts rejects cross-origin Origin/Referer

mismatches on cookie-authenticated /api/* mutations, and the login/signup

handlers validate the Origin header against the request host. Server-to-

server endpoints (Stripe webhook, inbound webhooks, email intake, SCIM) are

exempt — they authenticate via signature/bearer tokens, not cookies.

The vector store searches only global + the caller's tenant — cross-tenant

leakage is prevented by construction (and asserted in tests).

password (random non-login hash) and authenticate via the IdP.

2. Data classification

3. Incident response plan

the tamper-evident audit chain (a broken hash is an immediate signal).

integration toggle (webhooks, SSO, AI mode), isolate the tenant.

apply the fix behind the normal change/approval flow.

documents root cause and the fix; the audit chain provides the timeline.

4. Data retention

tenant (demo default; no auto-deletion). Production must set a policy aligned

with the customer's obligations (30-day demo default is documented in the

Admin → Compliance tab).

data; deleting a KB article removes its embedding (deleteBySource).

uploads (uploads live under the tenant's directory).

5. Change management

passing test suite (tsc, eslint, vitest).

docker compose up -d --build); destructive *data* changes (remediation,

fleet actions) always route through the human approval gate.

TriagoAI — Penetration Test Plan

A scoping document for an external penetration test ahead of SOC 2. This is a

plan, not the test itself; findings should be tracked as TriagoAI tickets

and remediated through the normal change/approval flow.

1. Scope

knowledge, infra, evidence pack, reports).

unauthenticated API surfaces (rate-limited login, error-boundary reporting).

audit export, demo/live tick endpoints.

node:22-alpine, SQLite/libsql + vectors.db sidecar, port mapping

80/3004 → 3000).

Out of scope

TriagoAI integration points, not the vendors.

2. Methodology

untrusted intake), broken authentication (JWT session forgery, SSO mock flow),

broken access control (tenant isolation, RBAC on admin-only endpoints,

approval gate bypass), SSRF via the SSO/webhook/LLM outbound calls, XSS in

rendered ticket/comment/agent content.

intakes (dedupe), rate-limiting, Content-Type/payload validation, and the

authorization: Bearer webhook-secret check.

tool-call allowlist enforcement (no execute_sql), and the self-critique /

human-approval gates.

3. Tools

probing against the SQLite backend.

and the audit chain implementation.

4. Schedule & frequency

change; a full re-test annually or on major architecture changes.

commit; the tamper-evident audit chain is exercised by automated tests.

5. Reporting format

Each finding is filed as an TriagoAI ticket and a report entry with:

CVSS or OWASP Risk Rating.

Findings are reviewed in the same CAB/approval flow as any change, and the

remediation commit is linked back to the ticket for the audit trail.

TriagoAI — Problems & Troubleshooting

Known issues, their causes and fixes, plus a record of past production

incidents and the postmortems that fixed them.

Quick reference

Postmortems

"error 2090111630" after creating a workspace (2026-08-16)

had set the session cookie secure: NODE_ENV === "production", but the local

Docker run is next start over plain HTTP. Safari drops Secure cookies

over HTTP, so the post-signup cookie never stored → redirect to /queues →

requireUser() threw UNAUTHENTICATED.

documented in .env.example/compose. Also added a self-diagnosing error

boundary (components/error-boundary.tsx + lib/error-diagnostics.ts) and the

ErrorReport table + admin "Errors" tab.

Stale code on port 3004 (2026-08-15)

(2) a Colima SSH mux held IPv4 *:3004 while a stray next dev bound IPv6

:3001; (3) docker compose build used a stale layer cache.

DeepSeek wraps output in code fences (2026-08-15)

and Remediation narrative; extractJson already handled fences for

triage/diagnosis.

Tenant deletion FK violation (2026-08-21)

after on-call shifts were seeded.

ON DELETE RESTRICT, blocking the tenant → user cascade.

20260821064108_oncall_shift_user_cascade).

Error reporting

Runtime errors surface through the self-diagnosing boundary and persist to

ErrorReport (admin → Errors). The tamper-evident audit chain doubles as a

detection signal — a broken hash marks the chain.

Escalation

For anything not here, run the repro locally, file it as an TriagoAI ticket, and

append the postmortem above once fixed (per the "update docs after each session"

convention in CONTEXT.md).