Observability

Ticket Board records what happened, in one place, with one schema. Domain mutations, auth failures, bot tool calls, token rotations, webhook deliveries — all land in the same audit_event table. The user-visible activity feed is a filtered projection of the same rows. Webhooks fan out from the same emission point. One request ID ties everything together.

This page is the operator-facing summary. The exhaustive spec lives in spec/07_observability.md. The one-table-with-category decision is ADR-0006; the separate tool-invocation table is ADR-0008.

Tenets

1. Every write is attributable. The audit log records who did what. There is no anonymous-mutation path; system-originated writes (sweepers, migrations) record actor_kind='system' explicitly.
2. Audit is the security log. Auth failures, token rotations, bot enable/disable, domain events — all in the same table. One stream, one query surface.
3. Activity is a projection, not a copy. The UI's activity feed is SELECT ... WHERE category = 'activity'. There is no second write path.
4. Operational visibility is single-host. Structured logs to stdout, Prometheus at /metrics, two health probes. No external trace store or log shipper in v1.
5. In-process webhook delivery is v1-only. The emitter contract is durable-queue-shaped so a v2 migration to Redis or a DB-backed outbox does not require touching services.

Audit log

Storage

audit_event is append-only forever. The DB role used by the app revokes UPDATE and DELETE on this table at migration time as belt-and-braces; the code path also has no update_audit_event or delete_audit_event function. The full schema is in Data model.

Key invariants:

• Append-only. No update path exists in repositories/.
• category ∈ {activity, audit_only}. activity rows feed the activity feed; audit_only rows are admin/security only.
• request_id correlates events emitted from the same HTTP/MCP request.
• actor_user_id is NULL only for system-originated rows; actor_kind is always populated.

Emission rules

Domain writes — after-commit hooks

The default path for everything except auth failures.

sequenceDiagram
    autonumber
    participant Svc as service
    participant UoW as unit_of_work
    participant Emitter as events.emitter
    participant DB

    Svc->>UoW: BEGIN
    Svc->>DB: INSERT ticket
    Svc->>Emitter: emit(ticket.created)
    Emitter->>UoW: enqueue(event) — not written yet
    Svc->>UoW: COMMIT
    UoW->>DB: COMMIT (domain tx)
    UoW->>Emitter: after_commit() — drain queue
    Emitter->>DB: INSERT audit_event(ticket.created) (separate short tx)
    Emitter->>Emitter: enqueue webhook deliveries

• On COMMIT: the after-commit hook drains the queue and writes audit rows in a separate short transaction. Webhook deliveries are enqueued in-process.
• On ROLLBACK: the hook does not run — the domain "didn't happen," and neither does its audit row. This is the correct behavior for, e.g., a ticket.created event whose transaction rolled back.

Auth-failure events — bypass the unit of work

auth.login.failure, auth.login.rate_limited, auth.bot.failure, auth.bot.write_attempt_in_read_mode: written immediately on a dedicated short-lived connection via audit_sink.write_now(event). The failure itself is the security signal — it must persist even if the surrounding request transaction rolls back.

audit.orphaned fallback

If a service raises an exception after emit(event) but before commit, the rollback discards the after-commit hooks and the trail is lost. As a defense, when an exception is about to propagate out of the unit of work and the emitter's queue is non-empty, the emitter writes one audit.orphaned row:

• event_type = 'audit.orphaned'
• category = 'audit_only'
• payload = { original_event_type, original_payload_redacted, original_target_kind, original_target_id }

The orphan write uses the same dedicated connection as write_now, so it survives the rollback. Redaction on the orphan payload is conservative — strings longer than 256 chars are truncated, and secret-keyed fields are replaced with <redacted>.

If the orphan write itself fails (DB unreachable), the emitter logs a structured audit.orphan_write_failed line to stdout with the original event type and request ID. The log is the last-resort trail.

Event taxonomy

The full catalog of event_type strings emitted in v1. Format is <domain>.<verb_past>. category is locked per row at write time, not inferred at read.

Auth domain

event_type	Category	Payload (key fields)
auth.login.success	audit_only	{ip, user_agent}
auth.login.failure	audit_only	{ip, user_agent, reason, username_attempted}
auth.login.rate_limited	audit_only	{ip, username_attempted, bucket}
auth.logout	audit_only	{session_id}
auth.session.expired	audit_only	{session_id, reason: 'idle'\|'absolute'}
auth.bot.success	audit_only	{token_id, token_prefix, surface: 'rest'\|'mcp'}
auth.bot.failure	audit_only	{token_prefix, surface, reason}
auth.bot.write_attempt_in_read_mode	audit_only	{surface, target_kind, target_id}
auth.password.changed	audit_only	{username} (with actor_kind='human')
auth.password.reset_via_cli	audit_only	{username} (with actor_kind='system')

Bot lifecycle

event_type	Category	Payload (key fields)
bot.created	audit_only	{bot_user_id, username, mode, project_ids}
bot.updated	audit_only	{bot_user_id, changes}
bot.enabled	audit_only	{bot_user_id}
bot.disabled	audit_only	{bot_user_id}
bot.token.created	audit_only	{bot_user_id, token_id, token_prefix}
bot.token.rotated	audit_only	{bot_user_id, old_token_id, new_token_id, new_token_prefix}
bot.token.revoked	audit_only	{bot_user_id, token_id}
bot.tool_invoked	audit_only	{tool_name} (with FK to tool_invocation_log.id)
bot.resource_read	audit_only	{uri} (MCP resources/read)

Ticket domain

Field-level changes (priority, due date, parent, label add/remove, assignee→unassigned) are not separate event types. They are folded into the umbrella ticket.updated event whose changes payload describes the diff per field. ticket.assigned is the lone exception — assignee changes also emit a dedicated row so per-assignee notifications stay easy to query.

event_type	Category	Payload (key fields)
ticket.created	activity	{display_id, type, title, state_id, priority, assignee_user_id, parent_ticket_id}
ticket.updated	activity	{display_id, changes: {field: {from, to}}} — covers priority, due_date, parent, labels, etc.
ticket.state_changed	activity	{display_id, from_state_id, from_state_name, to_state_id, to_state_name}
ticket.assigned	activity	{display_id, from_assignee_user_id, to_assignee_user_id} (either side may be NULL)
ticket.watcher_added / ticket.watcher_removed	activity	{display_id, user_id, subscribed_via}
ticket.relation_added / ticket.relation_removed	activity	{display_id, target_display_id, relation_type}
ticket.deleted	activity	{display_id, title}

Comment domain

event_type	Category	Payload (key fields)
comment.created	activity	{ticket_display_id, comment_id, body}
comment.updated	activity	{ticket_display_id, comment_id, body, previous_body}
comment.deleted	activity	{ticket_display_id, comment_id, body, edit_history} — full body preserved per ADR-0005

Wiki domain

event_type	Category	Payload (key fields)
wiki.created	activity	{slug, title, project_id}
wiki.updated	activity	{slug, changes: {field: {from, to}}}
wiki.deleted	activity	{slug, title}
wiki.render_failed	audit_only	{page_id, exc_class}

Sticky notes

event_type	Category	Payload (key fields)
sticky_note.created / .updated / .deleted	activity (project) / audit_only (personal)	{sticky_note_id, project_id?, body_preview}
sticky_note.converted	activity	{sticky_note_id, ticket_display_id}

Personal sticky notes do not appear in the project/workspace activity feed — other people would never see them anyway.

Project and config

event_type	Category	Payload (key fields)
project.created / .updated / .deleted	audit_only	{project_id, key, name, changes?}
workflow_state.created / .updated / .deleted	audit_only	{project_id, state_id, ...}
workflow_state.reordered	audit_only	{project_id, ordered_ids}
label.created / .updated / .deleted	audit_only	{project_id, label_id, ...}

Webhook domain

event_type	Category	Payload (key fields)
webhook.created / .updated / .deleted	audit_only	{webhook_id, url, event_types, changes?}
webhook.secret_rotated	audit_only	{webhook_id} (plaintext only in HTTP response)
webhook.delivery_succeeded	audit_only	{webhook_id, delivery_id, attempt, http_status}
webhook.delivery_failed	audit_only	{webhook_id, delivery_id, attempt, http_status, error}
webhook.delivery_retried	audit_only	{webhook_id, delivery_id, attempt} (manual retry triggered)

System

event_type	Category	Payload (key fields)
audit.orphaned	audit_only	{original_event_type, original_payload_redacted, original_target_kind, original_target_id}
system.sweeper_ran	audit_only	{sweeper, rows_purged} — one row per swept table per pass (sessions, idempotency, tool-log)

Retention

v1: forever. No retention sweeper runs against audit_event. Given the single-operator scale (one human + N bots producing hundreds to low-thousands of events/day), the table size is manageable for years.

The future hook point is partitioning the table by created_at month, then dropping or archiving partitions older than a configurable retention. Documented for v2; not implemented. The data-model invariant ("never deleted by app") still holds.

Activity feed projection

The activity feed and the audit log share a single endpoint (GET /audit). The frontend filters with category=activity for the activity feed and shows everything for the audit view.

The query:

SELECT *
FROM audit_event
WHERE workspace_id = :workspace_id
  AND category = 'activity'
  AND (:ticket_id IS NULL OR ticket_id = :ticket_id)
  AND (:project_id IS NULL OR project_id = :project_id)
ORDER BY created_at DESC
LIMIT :limit;

Indexes that cover this:

• INDEX (ticket_id, created_at DESC) WHERE ticket_id IS NOT NULL — per-ticket feed.
• INDEX (project_id, category, created_at DESC) WHERE project_id IS NOT NULL — per-project feed.
• INDEX (workspace_id, created_at DESC) — workspace-wide recent.

The frontend keeps a render registry keyed by event_type so that, for example, ticket.state_changed renders as <actor> moved <link:display_id> from <from_state> to <to_state>. Unknown event types render with a generic fallback (<actor> <event_type> <target>) so the UI stays forward-compatible.

Tool invocation log

Bot MCP tool calls write to two tables per call:

1. A tool_invocation_log row carrying the redacted args, result status, error message (if any), and duration. See Data model for the schema.
2. A lightweight audit_event row with event_type='bot.tool_invoked', category='audit_only', and tool_invocation_log_id pointing at the log row above. This is the discoverable fact-of-invocation in the audit stream.

Both writes happen after the domain commit. Failure modes:

Scenario	Result
Both writes succeed	Normal case. Audit stream shows bot.tool_invoked; detail is one FK hop away.
tool_invocation_log write fails	Pointer audit row is skipped (no orphan pointer). Emitter writes one audit.orphaned row with the redacted payload inline.
tool_invocation_log succeeds, audit_event fails	Log row stays (independently meaningful). Emitter writes one audit.orphaned row with tool_invocation_log_id in payload.

Retention

tool_invocation_log rows are purged after TOOL_INVOCATION_LOG_RETENTION_DAYS (default 90). The pointer audit_event row survives forever; when the log row is purged, the FK is set NULL — the fact-of-invocation persists; the click-through goes dead.

Operator UI

The /audit/tools route shows tool invocations in a dedicated view with their own columns (tool_name, duration_ms, result_status). Each row links back to the corresponding bot.tool_invoked audit row via the FK, placing the call in the broader audit timeline.

Webhook delivery

Subscription model

Webhooks are workspace-level subscriptions with a URL, an HMAC-SHA256 secret, and an event-type allowlist. See Data model.

Delivery worker

v1: in-process BackgroundTasks. No Redis, no Celery, no external queue.

sequenceDiagram
    autonumber
    participant Svc as service
    participant Emitter as events.emitter
    participant Dispatcher as webhook_dispatcher
    participant Worker as in-process worker
    participant Target as External URL
    participant DB

    Svc->>Emitter: emit(ticket.created)
    Emitter->>DB: INSERT audit_event(ticket.created) (after-commit)
    Emitter->>Dispatcher: enqueue_deliveries(event, audit_event_id)
    Dispatcher->>DB: SELECT webhook WHERE is_active AND event_types matches
    Dispatcher->>Worker: BackgroundTasks.add_task(deliver_one)

    loop attempts 1..3
        Worker->>Worker: build payload, HMAC-sign
        Worker->>Target: POST payload
        alt 2xx
            Target-->>Worker: 2xx
            Worker->>DB: INSERT webhook_delivery(attempt, delivered_at)
            Worker->>Emitter: emit(webhook.delivery_succeeded)
        else 4xx non-429
            Target-->>Worker: 4xx
            Worker->>DB: INSERT webhook_delivery(attempt, response_status)
            Worker->>Emitter: emit(webhook.delivery_failed, final=true)
        else 5xx, 429, network/timeout
            Target-->>Worker: error
            Worker->>DB: INSERT webhook_delivery(attempt, next_retry_at)
            Worker->>Emitter: emit(webhook.delivery_failed, will_retry=true)
            Worker->>Worker: sleep(backoff) then loop
        end
    end

Retry policy

• Attempts: 3 total (initial + 2 retries).
• Backoff: 1 s, 5 s, 30 s between attempts.
• Retry-eligible: HTTP 5xx, HTTP 429, network errors, timeouts.
• No-retry: HTTP 4xx other than 429. The target is telling us something we shouldn't ask again.

Each attempt writes a webhook_delivery row. The latest row's next_retry_at is set on transient failures and NULL on terminal success or terminal failure.

Signature scheme

HMAC-SHA256 over a canonical timestamp + "." + raw_body string, with a separate timestamp header for replay protection. This is the Stripe-style pattern.

Outbound headers:

Header	Value	Purpose
Content-Type	application/json; charset=utf-8	Standard JSON
X-Webhook-Id	<webhook.id>	Subscription identifier
X-Webhook-Event	<event_type>	E.g. ticket.created
X-Webhook-Delivery	<webhook_delivery.id>	Per-attempt ID (idempotency key for receiver)
X-Webhook-Timestamp	<unix_seconds>	For replay protection
X-Webhook-Signature	sha256=<hex> of HMAC-SHA256 over f"{timestamp}.{raw_body}"	Authenticity
User-Agent	ticket-board/<version>	Identify ourselves

Receiver verification (documented for integrators, not enforced by us):

1. Read X-Webhook-Timestamp. Reject if abs(now() - timestamp) > 5 minutes.
2. Compute expected = "sha256=" + hex(hmac_sha256(secret, f"{timestamp}.{raw_body}")).
3. Constant-time-compare against X-Webhook-Signature. Reject on mismatch.

Payload schema

{
  "id": "<webhook_delivery.id>",
  "type": "<event_type>",
  "occurred_at": "<audit_event.created_at ISO-8601 UTC>",
  "workspace_id": "<uuid>",
  "actor": { "user_id": "<uuid>", "kind": "human|bot|system", "username": "..." },
  "target": { "kind": "ticket|wiki_page|...", "id": "<uuid>", "display_id": "PROJ-123" },
  "payload": { },
  "request_id": "<uuid>"
}

The payload field mirrors the audit_event payload exactly. The audit row and the webhook payload are derivable from each other.

Failure resilience

Warning

If the FastAPI process dies between enqueue and delivery, pending in-process jobs are lost. This is the acceptable v1 trade-off.

• The webhook_delivery row is only written per attempt. If no attempt ran, no row exists. The receiver doesn't see a "lost" delivery; from their perspective, the event simply didn't fire.
• On startup, the worker does not replay missed events. We do not implement an outbox in v1.
• The audit_event row exists (it was written in the after-commit hook before enqueue, in a separate tx). The internal record of the event is intact; only the outbound notification is lost.

Future durable-queue migration shape (documented, not implemented):

1. Add a webhook_outbox table: (id, event_id, webhook_id, payload_jsonb, ready_at, attempts, status, ...).
2. Replace the in-process BackgroundTasks with a poll loop that pulls from webhook_outbox WHERE ready_at <= now() AND status = 'pending'.
3. Worker process can be the FastAPI process (poll-loop alongside requests), a separate Python worker (better isolation), or a Redis-backed queue (best isolation).
4. The emitter contract does not change: services call emit(event); the dispatcher writes to webhook_outbox instead of in-process enqueue.

The shape is mapped now so v1 doesn't bake in a non-migratable design.

Admin UI

/settings/webhooks lists subscriptions; each row links to /settings/webhooks/{id}/deliveries:

[Attempt #] [Time] [Event] [HTTP status] [Duration] [Error] [Actions]

Per-row actions:

• Retry: calls POST /webhooks/{id}/deliveries/{delivery_id}/retry. Synchronous best-effort.
• View payload: expands the original request body.

A delivery that has already succeeded cannot be retried (returns 409 resource.conflict).

Request correlation

A UUIDv7 request_id is generated in middleware for every inbound request (REST + MCP) and propagated everywhere:

• Stored on audit_event.request_id, tool_invocation_log.request_id, webhook_delivery.request_id.
• Bound into the structured logger's context for every log line emitted during the request.
• Carried into the webhook payload's request_id field.
• Surfaced in the X-Request-Id response header for client debugging and bug reports.

A user reporting a bug includes the request ID from a toast in the UI; the operator greps logs and audit for that ID and reconstructs the request's full timeline in one query.

Cross-surface correlation example

A bot creates a ticket via MCP. The full observability footprint:

sequenceDiagram
    autonumber
    participant Bot
    participant API as FastAPI
    participant Svc as ticket_service
    participant Emitter
    participant Dispatcher as webhook_dispatcher
    participant DB
    participant Target as Webhook receiver

    Bot->>API: POST /mcp/bot/<token>/mcp (create_ticket)
    Note over API: request_id = R1
    API->>Svc: dispatch
    Svc->>DB: INSERT ticket
    Svc->>Emitter: emit(ticket.created, request_id=R1)
    Svc->>Emitter: emit(bot.tool_invoked, request_id=R1)
    Svc->>DB: COMMIT (domain tx)

    Note over Emitter: After-commit hook
    Emitter->>DB: INSERT tool_invocation_log
    Emitter->>DB: INSERT audit_event(bot.tool_invoked, FK to log)
    Emitter->>DB: INSERT audit_event(ticket.created)
    Emitter->>Dispatcher: enqueue_deliveries

    Dispatcher->>Target: POST payload (X-Request-Id: R1)
    Target-->>Dispatcher: 200
    Dispatcher->>DB: INSERT webhook_delivery
    Dispatcher->>Emitter: emit(webhook.delivery_succeeded)

Operator inspecting "what happened" can:

• Query audit_event WHERE request_id = 'R1' → 3 rows (the domain event, the tool-invoked pointer, the delivery success).
• Click the tool-invoked row → drill to tool_invocation_log via the FK.
• Click the delivery success → see the webhook_delivery row with the full payload.
• Grep stdout for request_id=R1 → see all log lines from middleware, auth, service, dispatcher.

One ID ties it all together.

Structured logging

Library

structlog with JSON output in production and a pretty console renderer in dev. Selection via Settings.env at process startup.

Why structlog over alternatives:

• Native JSON output via processor chains.
• bind_contextvars ties the request_id (and other request-scoped tags) into every log line emitted during the request without threading them through every function.
• Plays nicely with logging.getLogger(__name__) — third-party logs (SQLAlchemy, Uvicorn, Alembic) still get JSON-rendered.

Standard fields

Every line carries:

Field	Source	Always present?
timestamp	TimeStamper(fmt="iso", utc=True)	yes
level	log call	yes
event	log message	yes
request_id	bind_contextvars from RequestContextMiddleware	yes for in-request logs

Other fields (actor_user_id, actor_kind, workspace_id, http_method, http_path, http_status, duration_ms) appear only on log lines emitted from sites that bind them explicitly — they are not threaded through every line by middleware. The instrumentator emits its own HTTP request/response telemetry to Prometheus rather than to structlog.

Redaction

Two redaction processors run as part of the structlog chain:

1. Secret-keyed fields: any JSON field whose key matches password, token, secret, api_key, private_key (case-insensitive) is replaced with <redacted>. Same allowlist as tool_invocation_log.
2. MCP path tokens: paths matching /mcp/bot/<token>/... are normalized to /mcp/bot/<redacted>/... in any field where they might appear.

The matching Caddy-side redaction is described in Auth and identity.

Where logs go in v1

Stdout only. Docker Compose captures stdout; the operator views with docker compose logs -f. No remote shipping, no log aggregator (Loki, ELK, CloudWatch). The JSON-renderer output is forward-compatible with any structured-log shipper that might appear later.

Metrics

/metrics exposes Prometheus-format metrics via prometheus-fastapi-instrumentator. Lowest-friction option: a single import and one-liner setup produces request count + latency histograms out of the box.

v1 metrics

Two HTTP metrics come from prometheus-fastapi-instrumentator out of the box; two webhook metrics are defined in events/webhook_dispatcher.py. The remaining four metrics are planned, not implemented — they're listed so dashboard authors know what to expect when they ship.

Metric	Type	Labels	Status	Purpose
http_requests_total	counter	method, route, status	shipped	Request count by route
http_request_duration_seconds	histogram	method, route	shipped	Latency per route
webhook_deliveries_total	counter	webhook_id, event_type, result	shipped	Outbound delivery health
webhook_delivery_duration_seconds	histogram	webhook_id, event_type	shipped	Delivery latency
mcp_tool_invocations_total	counter	tool_name, result	planned	Bot tool usage
audit_events_total	counter	event_type, category	planned	Emission rate; canary for "did the firehose stop?"
db_pool_size / db_pool_in_use	gauge	(none)	planned	Connection pool health

route is the templated path (e.g. /api/v1/tickets/{id_or_display_id}), not the substituted URL. The instrumentator handles this via FastAPI's route table, which prevents label-cardinality explosion.

Endpoint protection

/metrics is protected by either:

• Localhost-only by default (the route checks request.client.host against a configured allowlist).
• Basic auth as an alternative (METRICS_USERNAME, METRICS_PASSWORD), enabled when set.

Dashboards

Out of scope for v1. The starter dashboard set is documented in operator notes (future deliverable):

• API health: requests/sec, p50/p95/p99 latency by route, 5xx rate.
• Webhook health: delivery success rate per subscription, mean delivery latency, retry rate.
• Bot activity: tool invocations/min by bot, error rate per tool.

Health checks

Two endpoints, both unauthenticated and intentionally minimal.

/healthz — liveness

• Returns 200 {"status": "ok"} if the process is responsive.
• Does NOT touch the database. A DB outage should not flap liveness — the process is still alive, just degraded.

/readyz — readiness

• Returns 200 {"status": "ready", "db": "ok", "settings_loaded": true} if:
• DB is reachable (SELECT 1 in under 500 ms).
• The session-maker is configured.
• Returns 503 with a structured body if either check fails.

Startup invariants (e.g. cookie_secure=true in prod) are asserted once in lifespan.py before the listener opens. They are not re-checked on every probe — if /readyz responds at all, the process passed them at boot.

Caddy uses /healthz for "is the container up." Docker Compose's healthcheck: block uses /readyz for "is the container ready to serve." Both endpoints are documented in the OpenAPI schema under a health tag.

Tracing

Out of scope for v1.

The natural injection site is the request-context middleware, alongside the request_id generation. A v2 PR would parse W3C traceparent on inbound, propagate to downstream HTTP calls (webhook deliveries), and attach to audit_event via a new trace_id column. OpenTelemetry's opentelemetry-instrumentation-fastapi handles the middleware side.

Retention sweeper

A single background job handles three retention concerns on a shared schedule (SWEEPER_INTERVAL_MINUTES, default 60):

Table	Purge condition
session	expires_at < now(). Idle-expired rows are caught here once they cross absolute.
idempotency_record	expires_at < now() (IDEMPOTENCY_KEY_TTL_HOURS, default 24).
tool_invocation_log	created_at < now() - TOOL_INVOCATION_LOG_RETENTION_DAYS (default 90). FK on the pointer audit_event row is set NULL in the same pass.

Each pass emits one system.sweeper_ran event per table with {sweeper, rows_purged}.

Failure modes:

• A failed sweeper pass means rows accumulate until the next successful run. The worst outcome is bloated tables, not data corruption.
• Sweeper errors log a structured line and increment a metric. The metric is a good alerting hook (no rows purged in 24 hours = investigate).

Failure modes table

The "what if X breaks" matrix. Each row: a single failure, the user-visible effect, and the operational response.

Cause	User-visible effect	Operational response
Postgres unreachable on inbound request	503 server.unavailable + toast; FE may show stale TanStack Query cache	Operator restarts DB; /readyz is failing during outage.
Postgres unreachable on after-commit audit write	Domain write succeeded; audit row written via dedicated connection on next attempt	If dedicated connection also fails, emitter logs audit.orphan_write_failed.
Audit write fails after domain commit	Domain change is live but audit row missing; emitter writes audit.orphaned	Orphan rows surface in /audit filtered by event_type='audit.orphaned'.
Webhook target unreachable	webhook.delivery_failed (will_retry=true) audit events; up to 3 attempts	Operator sees failures in /settings/webhooks/{id}/deliveries; can hit Retry.
Webhook target returns 4xx (non-429)	webhook.delivery_failed (will_retry=false) immediately	Operator inspects target; fixes config; manual retry.
FastAPI process crashes mid-webhook-delivery	Pending in-process queue items are lost. Already-written webhook_delivery rows persist. No replay on startup.	Operator monitors webhook_deliveries_total; v2 outbox prevents this.
Sweeper job fails	Tables grow until next successful run; no functional impact	Sweeper logs an error; metric ticks; operator investigates.
tool_invocation_log write fails	Pointer audit row skipped; emitter writes audit.orphaned with inline redacted payload	Orphan rows in /audit surface the gap.
request_id middleware not run (config bug)	All audit rows for that request have NULL request_id	CI test asserts middleware runs; on prod, absent request_id is detectable as a metric anomaly.
Login attempt floods (DoS)	Per-IP and per-username buckets hit; legitimate user temporarily locked out	auth.login.rate_limited events; operator clears bucket via restart in v1.
Webhook secret leaked	Receiver sees signature-valid replay attempts	Operator rotates via PATCH /webhooks/{id}?rotate_secret=true; emits webhook.secret_rotated.
Bot token leaked	Attacker can impersonate bot	Operator revokes via POST /bots/{id}/tokens/revoke. Audit log captures any actions taken (greppable by token_prefix).
Reverse proxy fails to redact /mcp/bot/<token>/...	Token appears in upstream access logs	Operator-runbook responsibility; spec mandates redaction at proxy.
Wiki render exception	Page shown as raw markdown wrapped in <pre>; wiki.render_failed audit event	Operator investigates; fix is auto-picked-up on next read.
Clock skew between server and client	ETags still work; webhook receivers may reject stale X-Webhook-Timestamp	Document the 5-min replay window; operators sync NTP.

For the full failure-mode matrix and severity tagging, see spec/07_observability.md.

Operator dashboards (frontend)

Recap of what the human operator sees in the UI:

FE route	Backend surface	Shows
/audit	GET /audit	Full audit log with filters
/audit/tools	GET /audit/tools	Per-call tool-invocation log; bot/tool filters
/ (dashboard widget)	GET /audit?category=activity&limit=20	Recent activity feed
/projects/[key] (widget)	GET /audit?project_id=...&category=activity	Per-project activity
/tickets/[displayId] (feed)	GET /audit?ticket_id=...&category=activity	Per-ticket activity
/settings/webhooks	GET /webhooks	Subscription list
/settings/webhooks/{id}/deliveries	GET /webhooks/{id}/deliveries	Delivery log per subscription

/metrics and /healthz are for operator tooling, not the UI. See the Operator guide for the audit-log walkthrough.

What this page does not cover

• The wire shape of GET /audit and GET /audit/tools (filters, cursor format) — see the REST API reference.
• Specific Prometheus / Grafana dashboard JSON — devops-expert deliverable, in the operator runbook.
• Log shipping pipeline — out of v1 scope.
• FE component implementation for the audit / delivery viewers — spec/06_frontend.md.