Troubleshooting#

The top issues you'll hit running Ticket Board, with the exact log greps, curl probes, and SQL queries to triage them. Each entry follows the same shape: symptom, what's happening, how to confirm, how to fix.

If you've checked everything here and the problem persists, the request id surfaced by the UI toast (or the X-Request-Id response header) is the fastest debug primitive — paste it into /audit?request_id=<uuid> and grep docker compose logs backend for the same id.

0. Triage primitives#

Before anything else:

# Is the stack up?
docker compose ps

# Are services healthy? (look for "healthy" in the State column)
docker compose ps --format json | jq '.[] | {Service, State, Health}'

# Quick readiness check.
curl -s http://localhost/readyz | jq

# Recent backend logs.
docker compose logs --tail=50 backend

If any of those four reveal an obvious problem (a container restarting, /readyz returning 503, repeating exceptions in the logs), start there.

Symptom#

You enter correct credentials, get redirected to the dashboard, then immediately bounced back to /login with a "Session expired" toast.

What's happening#

Almost always one of:

COOKIE_SECURE=true while running plain HTTP (no TLS). The browser silently drops the cookie because Secure is set.
A reverse proxy or browser extension stripping the Set-Cookie header.
A mismatched SameSite setup (single-origin is required; backend on a different host than the FE breaks the cookie).

Confirm#

# Look at the Set-Cookie header on a real login.
curl -i -s -X POST http://localhost/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"changeme-please"}' \
  | grep -i set-cookie

If you see Secure; in the cookie attributes but you're using http://, the browser is dropping it.

docker compose logs backend | grep -E 'startup|COOKIE_SECURE'

Should reveal whether the backend booted with secure cookies.

Fix#

In your .env, for plain-HTTP local dev:

ENV=dev
COOKIE_SECURE=false

Then restart:

docker compose down && docker compose up -d

For production with Caddy terminating TLS:

ENV=prod
COOKIE_SECURE=true

The backend refuses to start with ENV=prod + COOKIE_SECURE=false at startup — see lifespan.py. If you hit that block, the startup log will tell you.

Single origin matters

Caddy puts the frontend and the backend on the same host and port. If you expose the backend on a different hostname (e.g. api.example.com vs app.example.com), the cookie's SameSite=Lax will not survive cross-site fetches. Keep the single-origin topology unless you're ready to switch to SameSite=None; Secure and accept the trade-offs.

2. Bot returns 401 `auth.invalid_credentials`#

Symptom#

A bot that worked yesterday now returns 401 on every REST call or every MCP message.

What's happening#

One of:

Token rotated. Rotation in v1 is instantaneous; the old token is rejected the moment a new one is issued.
Token revoked. Same effect — revoked_at is set, the row is is_active=false.
Bot deactivated. user.is_active = false rejects every credential the bot might present.
Token never set on the client. Configuration drift; the bot is sending no X-API-Key or an empty one.

Confirm#

# Pull recent auth failures from the audit log.
curl -s "http://localhost/api/v1/audit?event_type=auth.bot.failure&limit=10" \
  -H "Cookie: tb_session=$YOUR_SESSION" | jq '.data'

The payload.token_prefix field tells you which token is failing and payload.reason says why (unknown, revoked, hash_mismatch).

# Inspect the bot's state.
curl -s "http://localhost/api/v1/bots/$BOT_ID" \
  -H "Cookie: tb_session=$YOUR_SESSION" | jq '{username, is_active, mode, active_token_prefix}'

If is_active=false, the bot is deactivated.

Fix#

Cause	Fix
Token rotated	Update the bot client's `X-API-Key` (and MCP URL) with the new plaintext from the rotation response. If you didn't capture it, rotate again.
Token revoked, want it back	`POST /bots/{id}/tokens/rotate` to issue a fresh token.
Bot deactivated	`PATCH /bots/{id}` with `{"is_active": true}`.
Token never set	Fix the client config. Don't forget the `tkb_` prefix — the full token is what `X-API-Key` expects.

See Bots → Token rotation.

3. Bot returns 403 `auth.forbidden` (`reason=project_not_in_allowlist`)#

Symptom#

A read-mode or write-mode bot can authenticate but every project-scoped call returns 403.

What's happening#

The bot's bot_project_allowlist does not include the project id it's calling against. The service layer rejects with Forbidden(reason="project_not_in_allowlist").

Confirm#

curl -s "http://localhost/api/v1/bots/$BOT_ID" \
  -H "Cookie: tb_session=$YOUR_SESSION" | jq '.project_ids'

Compare against the project id the bot is trying to reach.

# Recent forbidden hits for this bot.
curl -s "http://localhost/api/v1/audit?actor_user_id=$BOT_ID&event_type=auth.bot.write_attempt_in_read_mode&limit=10" \
  -H "Cookie: tb_session=$YOUR_SESSION" | jq '.data'

Fix#

Add the project to the allowlist:

curl -s -X PATCH "http://localhost/api/v1/bots/$BOT_ID" \
  -H "Content-Type: application/json" \
  -H "X-Requested-By: web" \
  -b "tb_session=$YOUR_SESSION" \
  -d '{"project_ids": ["01H...", "01J..."]}'

The list is a full replacement, not a partial update — include every project the bot needs, not just the new one.

See Bots → Project allowlist.

4. Bot returns 403 `auth.read_only_bot`#

Symptom#

A mode=read bot calls a write endpoint or MCP tool and gets 403.

What's happening#

Read-only mode rejects writes at two layers:

MCP dispatch (defense-in-depth): a read bot calling create_ticket or any other write-scope tool is rejected before the tool body runs. Audited as auth.bot.write_attempt_in_read_mode.
Service layer: current_bot_writer dependency rejects read-only bots on every write endpoint.

Confirm#

# Is the bot read-only?
curl -s "http://localhost/api/v1/bots/$BOT_ID" \
  -H "Cookie: tb_session=$YOUR_SESSION" | jq '.mode'

If it returns "read", the bot is read-only.

Fix#

Promote the bot to write mode:

curl -s -X PATCH "http://localhost/api/v1/bots/$BOT_ID" \
  -H "Content-Type: application/json" \
  -H "X-Requested-By: web" \
  -b "tb_session=$YOUR_SESSION" \
  -d '{"mode": "write"}'

Or change the bot client to call read-only tools only. If you're unsure whether you want this bot writing, the safer default is to keep it read-only and have a second write-mode bot for the specific workflow.

See Bots → Read vs write mode.

5. Webhook deliveries keep failing#

Symptom#

/settings/webhooks/{id}/deliveries shows failed attempts piling up; the receiver isn't getting the data or is rejecting it.

What's happening#

Common causes:

Signature mismatch. Receiver computes the signature differently — wrong secret, re-serialized body, wrong concatenation, lower-case sha256= mismatch.
Replay window mismatch. Receiver rejects timestamps older than its own window (or the receiver's clock is wrong).
Receiver returns 4xx (non-429). Treated as terminal; no retry.
Network unreachable. Receiver host or port is unreachable from the Compose network.

Confirm#

# Pull recent failed deliveries.
curl -s "http://localhost/api/v1/webhooks/$WEBHOOK_ID/deliveries?status=failure&limit=10" \
  -H "Cookie: tb_session=$YOUR_SESSION" | jq '.data[] | {attempt_number, response_status, error, created_at}'

The error field carries transport errors; response_status carries the HTTP code if a connection succeeded. The full payload is available via "View payload" in the UI or GET /webhooks/{id}/deliveries row detail.

If the receiver is local:

# Can the backend container reach the receiver?
docker compose exec backend curl -sv http://host.docker.internal:9999/hook

If host.docker.internal is unreachable from inside the container, that's the issue (Linux hosts sometimes need --add-host=host.docker.internal:host-gateway in compose).

Fix#

Cause	Fix
Signature mismatch	Verify the receiver computes `hmac_sha256(secret, f"{timestamp}.{raw_body}")` against the raw body bytes (not re-serialized). See Webhooks → Verifying on the receiver.
Wrong secret	Rotate via `PATCH /webhooks/{id}?rotate_secret=true`; update the receiver.
Clock skew	NTP-sync the receiver host. The default replay window is 5 minutes.
4xx response	Fix the receiver to accept the payload (often a content-type or routing issue). Manual retry once the receiver is fixed.
Network unreachable	Confirm the URL is reachable from the backend container, not just from your host.

After fixing, use the Retry button on a failed delivery row to re-fire that specific payload without waiting for the next event.

See Webhooks → Retry policy and Webhooks → The delivery log.

6. Ticket transition fails: 409 `etag_mismatch`#

Symptom#

You PATCH a ticket and get back 409 resource.conflict (reason=etag_mismatch). The UI shows a "Someone else edited this. Reload?" modal.

What's happening#

GETs return an ETag header. PATCHes that include If-Match: <etag> are rejected with 409 if the row has been updated since the ETag was issued — protection against silent overwrites in a human-vs-bot race.

Confirm#

# Get the current ETag.
curl -i -s "http://localhost/api/v1/tickets/INFRA-42" \
  -H "Cookie: tb_session=$YOUR_SESSION" | grep -i etag

Compare against the If-Match value your client sent.

Fix#

In the UI: click "Reload" in the modal. The page refetches, you re-apply your changes.
In a script: re-GET the ticket, capture the new ETag, retry the PATCH with the fresh If-Match.
To bypass: omit the If-Match header. Your write will succeed but you may silently overwrite a concurrent edit. Useful for transition or label toggles where the conflict is acceptable; not useful for body edits where data loss matters.

See Tickets → ETags and concurrent edits.

7. Workflow state delete fails: 409 `state_in_use`#

Symptom#

You try to delete a workflow state and get 409 resource.conflict (reason=state_in_use).

What's happening#

The state still has non-deleted tickets in it. Deleting it would leave those tickets in an invalid state (state_id pointing at a deleted row).

Confirm#

# How many tickets are currently in this state?
curl -s "http://localhost/api/v1/projects/$PROJECT_ID/tickets?state_id=$STATE_ID&limit=1" \
  -H "Cookie: tb_session=$YOUR_SESSION" | jq '.page'

If has_more: true or the data array is non-empty, there are tickets blocking the delete.

Fix#

Move the tickets to another state first:

In the UI, drag every card off the column. Or use the Bulk move action on the backlog view (filter by state, multi-select, Move to).
From the API, PATCH each ticket's transition:

curl -s -X POST "http://localhost/api/v1/tickets/$TICKET_ID/transition" \
  -H "Content-Type: application/json" \
  -H "X-Requested-By: web" \
  -b "tb_session=$YOUR_SESSION" \
  -d '{"to_state_id": "$NEW_STATE_ID"}'

Once the state is empty, retry the delete.

If the column has many soft-deleted tickets in it, those don't block the delete (the constraint is on non-deleted only) but they still surface in ?include_deleted=true views. That's fine; they continue to reference the now-deleted state, which is acceptable for forensic reads.

See Projects → Workflow states.

8. Wiki edit fails: 422 `validation.invalid_field` (body too long)#

Symptom#

Saving a wiki page returns 422 validation.invalid_field with details.errors[0].type = "string_too_long".

What's happening#

The body cap is 1 MiB (1,048,576 characters). The Pydantic schema rejects anything longer.

Confirm#

# How big is your body?
wc -c /path/to/your/markdown.md

If > 1,048,576, you're over.

Fix#

Pages this large are not documentation; they're data. Options in order of "least surgery":

Split the page. Move sections to linked wiki pages with [[wikilinks]]. The linked structure is more navigable than a 2 MB monolith.
Move data into tickets. If the page is mostly tabular content, that's probably ticket bodies or label-organized backlog.
External storage. If it's truly a doc that needs to be this big (a long-form runbook, an archived RFC), keep it in a Git repo or object store and reference it from a short wiki page.

The cap is intentional: the markdown parser is fast but not free, and the sanitizer's memory grows with HTML size. An OOM in the FastAPI process is much worse than a 422.

See Wiki → Body size.

9. `/readyz` returns 503#

Symptom#

curl http://localhost/readyz returns 503 with a structured body indicating one of the readiness checks failed.

What's happening#

/readyz checks:

DB reachable (SELECT 1 in <500 ms).
Settings loaded (hard-coded true once the process is up).

A 503 means the DB probe failed.

The ENV=prod + COOKIE_SECURE=true invariant is enforced at app startup in lifespan.py — if the assertion fails, the process refuses to start at all. There is no /readyz response in that case because there is no live process to answer. Check docker compose logs backend for the startup-time assertion error instead.

Confirm#

# What's the readiness body actually saying?
curl -s http://localhost/readyz | jq

# Is Postgres up?
docker compose ps postgres

# Recent backend logs.
docker compose logs --tail=100 backend | grep -E 'startup|readyz|error|ERROR'

Fix#

Failure	Cause	Fix
`db: error`	Postgres unreachable, migrations not run, DATABASE_URL wrong	`docker compose logs postgres`. If Postgres is alive, check `DATABASE_URL` matches the Compose service name (`postgres:5432`, not `localhost:5432`).
Process never comes up	`ENV=prod` without `COOKIE_SECURE=true`, malformed `.env`, missing `SECRET_KEY`	These are startup assertions; the process exits before serving. Check `docker compose logs backend` for the exact assertion. Fix the env var and restart. `/readyz` will start answering once the process boots.

/healthz is a separate, lighter probe — it returns 200 as long as the FastAPI process is responsive, without touching the DB. A DB outage will make /readyz fail but /healthz succeed. The Compose healthcheck: block uses /readyz for orchestration; a reverse proxy in front would use /healthz for liveness.

See spec/07_observability.md#health-checks.

10. MCP URL appears in access logs#

Symptom#

You grep the Caddy access log (or some other upstream log) and see /mcp/bot/tkb_<full_token>/... lines. That token is a bearer credential. It must never appear verbatim in any log.

What's happening#

Either:

The Caddy redaction is not configured (the included Caddyfile ships with it correctly).
An additional upstream proxy is logging before the Caddy redaction takes effect.
The backend's structlog middleware isn't running (config bug).

Confirm#

# Look for unredacted MCP URLs in Caddy logs.
docker compose logs proxy | grep -E '/mcp/bot/tkb_'

# Look in backend logs.
docker compose logs backend | grep -E '/mcp/bot/tkb_'

# The greppable signature: any tkb_ followed by 32 chars of base32.
docker compose logs | grep -E 'tkb_[a-z2-7]{32}'

If the third grep returns non-empty matches in any log surface, you have a leak.

Fix#

Check the Caddyfile redaction is in place — the included Caddyfile has a log { format filter { fields { request>uri regexp ... } } } block that rewrites /mcp/bot/<token> to /mcp/bot/<redacted>. If you've modified it, restore the redaction.
Check upstream proxies. If you're terminating TLS at an upstream load balancer (not the included Caddy), apply the same redaction at the LB. For nginx:

map $request_uri $redacted_uri {
    ~^(/mcp/bot/)[^/]+(.*)$  $1<redacted>$2;
    default                  $request_uri;
}

log_format  redacted  '$remote_addr - [$time_local] '
                      '"$request_method $redacted_uri $server_protocol" '
                      '$status $body_bytes_sent';

Rotate every leaked token immediately. If a token has appeared in a log that's been shipped off-host or backed up, treat it as compromised. POST /bots/{id}/tokens/rotate for each affected bot. Update clients with new plaintext.
Audit fallout. Check auth.bot.success events for the leaked token's prefix:

curl -s "http://localhost/api/v1/audit?event_type=auth.bot.success&limit=200" \
  -H "Cookie: tb_session=$YOUR_SESSION" \
  | jq '.data[] | select(.payload.token_prefix == "<the_prefix>")'

Every row is a successful use of the leaked token. Cross-reference each request_id against subsequent events to see what was done.

See spec/03_mcp_integration.md#security-notes and spec/04_auth_and_identity.md#security-checklist.

Bonus: useful greps#

A handful of greps the operator ends up running often:

# Auth failures (any kind).
docker compose logs backend | grep -E 'auth\.(login\.failure|bot\.failure|session\.expired)'

# Orphaned audit rows (defense net firing — investigate).
docker compose logs backend | grep -E 'orphan_write_failed|audit\.orphaned'

# Sweeper activity.
docker compose logs backend | grep -E 'system\.sweeper_ran'

# Any 5xx response.
docker compose logs backend | grep -E 'http_status":5'

# All log lines from one request id.
docker compose logs backend | grep "request_id=<UUID>"

# Wiki render failures.
docker compose logs backend | grep -E 'wiki\.render_failed'

And the canonical "audit row for one request" SQL, run from inside the postgres container:

docker compose exec postgres psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "
SELECT created_at, event_type, actor_kind, target_kind, target_id
FROM audit_event
WHERE request_id = '<UUID>'
ORDER BY created_at;
"

Where to file a real bug#

If you've reproduced an issue that none of the entries here covers — and it's not a configuration or credential issue:

Capture the request id from the UI toast or X-Request-Id response header.
Capture the audit rows: GET /api/v1/audit?request_id=<uuid>.
Capture the backend logs for that request: docker compose logs backend | grep <uuid>.
File an issue against the repo with all three.

The request id is the load-bearing piece. With it, every other surface (audit, logs, webhook deliveries) is one query away.

Troubleshooting#

0. Triage primitives#

1. Login fails: "Session expired. Please log in."#

Symptom#

What's happening#

Confirm#

Fix#

2. Bot returns 401 auth.invalid_credentials#

Symptom#

What's happening#

Confirm#

Fix#

3. Bot returns 403 auth.forbidden (reason=project_not_in_allowlist)#

Symptom#

What's happening#

Confirm#

Fix#

4. Bot returns 403 auth.read_only_bot#

Symptom#

What's happening#

Confirm#

Fix#

5. Webhook deliveries keep failing#

Symptom#

What's happening#

Confirm#

Fix#

6. Ticket transition fails: 409 etag_mismatch#

Symptom#

What's happening#

Confirm#

Fix#

7. Workflow state delete fails: 409 state_in_use#

Symptom#

What's happening#

Confirm#

Fix#

8. Wiki edit fails: 422 validation.invalid_field (body too long)#

Symptom#

What's happening#

Confirm#

Fix#

9. /readyz returns 503#

Symptom#

What's happening#

Confirm#

Fix#

10. MCP URL appears in access logs#

Symptom#

What's happening#

Confirm#

Fix#

Bonus: useful greps#

Where to file a real bug#

2. Bot returns 401 `auth.invalid_credentials`#

3. Bot returns 403 `auth.forbidden` (`reason=project_not_in_allowlist`)#

4. Bot returns 403 `auth.read_only_bot`#

6. Ticket transition fails: 409 `etag_mismatch`#

7. Workflow state delete fails: 409 `state_in_use`#

8. Wiki edit fails: 422 `validation.invalid_field` (body too long)#

9. `/readyz` returns 503#