Kill Switch Recovery

The kill switch is an emergency stop that blocks all agent traffic instantly. This runbook covers activation, investigation, and recovery.

Activation

Via dashboard

Settings → Kill Switch → Activate

The kill switch can be scoped:

| Scope | Effect | |---|---| | All | Blocks all agent traffic across all connectors | | Connector (e.g., github) | Blocks all traffic to a specific connector | | Agent (e.g., DevOps Bot) | Blocks all traffic from a specific agent |

Via CLI

# Activate (all traffic)
npx tf kill-switch --activate
 
# Activate (scoped to a connector)
npx tf kill-switch --activate --scope github
 
# Activate (scoped to an agent)
npx tf kill-switch --activate --scope "DevOps Bot"

Via API

curl -X POST http://localhost:3000/api/v1/kill-switch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"active": true, "scope": "all"}'
Warning
The kill switch takes effect immediately. All matching in-flight requests are terminated. Pending approvals are denied.

What happens when the kill switch is active

| Component | Behavior | |---|---| | Gateway (proxy mode) | All matching requests return 403 with X-TameFlare-Kill-Switch: active header | | Control plane (SDK mode) | All matching action requests return denied with reason kill_switch_active | | Dashboard | Kill switch status shown in Settings and header health indicator | | Audit log | kill_switch.activated event recorded with scope and activating user | | Pending approvals | Matching pending approvals are auto-denied |

Investigation runbook

When you activate the kill switch, follow this process to investigate and recover:

1. Confirm the kill switch is active

npx tf status
# Should show: Kill switch: ACTIVE (scope: all)

2. Review recent traffic

# Last 50 traffic entries
npx tf logs --last 50
 
# Or via dashboard: Traffic page with filters

Look for:

  • Unusual action types (e.g., github.repo.delete, stripe.transfer.create)
  • High request volume from a single agent
  • Actions that were denied by policy but retried repeatedly
  • Requests to unexpected domains

3. Check the audit log

# Via dashboard: Audit Log page
# Filter by event type, agent, or time range
# Export to CSV for offline analysis

Key events to look for:

  • action.denied — what was the agent trying to do?
  • action.allowed — were any dangerous actions allowed before the kill switch?
  • approval.responded — were any approvals granted that shouldn't have been?

4. Identify the root cause

| Symptom | Likely cause | Action | |---|---|---| | Agent making unexpected API calls | Prompt injection or hallucination | Review agent prompts, add policy rules | | High volume of requests | Agent in a loop | Fix agent code, add rate limit policy | | Requests to unknown domains | Agent discovered new APIs | Add connector or block domain | | Approved actions that shouldn't have been | Approver error or social engineering | Review approval policies, restrict approver group |

5. Fix the issue

Before deactivating the kill switch:

  • Update policies to prevent the problematic behavior
  • Revoke or suspend the affected agent if needed
  • Rotate credentials if you suspect credential compromise
  • Add stricter permissions for the affected connector

6. Deactivate the kill switch

# Via CLI
npx tf kill-switch --deactivate
 
# Via dashboard: Settings → Kill Switch → Deactivate
 
# Via API
curl -X POST http://localhost:3000/api/v1/kill-switch \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"active": false}'
Note
Only users with the owner role can activate or deactivate the kill switch.

7. Verify recovery

# Check status
npx tf status
# Should show: Kill switch: INACTIVE
 
# Run a test action
npx tf run --name "Test Bot" curl https://api.github.com/zen
# Should succeed if permissions are configured

Notification options

TameFlare can notify you when the kill switch is activated:

| Method | Configuration | |---|---| | Slack | Set SLACK_BOT_TOKEN + SLACK_CHANNEL_ID. Kill switch events are sent to the configured channel. | | Webhook | Set webhook_url on action requests. Kill switch denials include the reason in the webhook payload. | | Dashboard | Kill switch status is shown in the header health indicator with auto-refresh (10s). | | Audit log | kill_switch.activated and kill_switch.deactivated events are always recorded. |

Preventing false alarms

To avoid unnecessary kill switch activations:

  • Use scoped kill switches — block one agent or connector instead of everything
  • Use monitor enforcement level first — log without blocking to understand traffic patterns
  • Set up approval workflows — require human approval for high-risk actions instead of blocking everything
  • Review policies regularly — overly permissive policies lead to unexpected behavior

Next steps