Control-agent blocks on synchronous worker spawns, causing complete unresponsiveness

[vltbaudbot]

Summary

The control-agent can become completely unresponsive for extended periods (20+ minutes observed) when worker spawn commands block the main event loop. This creates a critical outage where all incoming messages are queued but never processed.

Problem Statement

Symptom: Control-agent stops processing inbox messages while bridge continues to receive and forward them.

Root Cause: The control-agent executes pi session spawn commands synchronously via the bash tool. If the spawn process hangs or takes a long time, the control-agent cannot process any other messages until it completes.

Impact:

• Complete loss of responsiveness to incoming requests
• Messages queue up with no acknowledgment or processing
• Users experience total silence from the system
• No automatic detection or recovery
• Requires manual intervention to restore service

Reproduction

1. Have control-agent with message queue
2. Execute worker spawn: pi session spawn --name "dev-agent-..." --skill dev-agent ...
3. If spawn fails/hangs (e.g., missing API key, config error):
   • Spawn retries multiple times
   • Each retry takes 30-60 seconds
   • Total: 5-10 minutes of blocking
4. Meanwhile: All incoming messages queue up unprocessed

Proposed Solutions

1. Non-Blocking Worker Spawns (Critical Priority)

Change: Always background worker spawns and verify separately

Before (blocks):

pi session spawn --name "dev-agent-..." --skill dev-agent --model X 2>&1 | tail -5

After (non-blocking):

# Spawn in background
(pi session spawn \
  --name "dev-agent-..." \
  --skill dev-agent \
  --model X \
  > /tmp/spawn-${WORKER_ID}.log 2>&1 &)

# Wait briefly for socket
sleep 3

# Verify spawn succeeded
if [ -S ~/.pi/session-control/${WORKER_NAME}.sock ]; then
  echo "Worker spawned successfully"
  # Send task immediately
  send_to_session --sessionName "${WORKER_NAME}" --message "..."
else
  echo "Worker spawn failed - check logs"
fi
# Continue processing inbox immediately!

Implementation suggestions:

• Update control-agent skill documentation with non-blocking spawn pattern
• Add helper function/script for safe worker spawning
• Warn in dev-agent docs about spawn timing

2. Default Timeouts on Bash Tool (Critical Priority)

Problem: Bash commands can run indefinitely

Solution: Add configurable timeout with reasonable default

interface BashToolOptions {
  command: string;
  timeout?: number; // milliseconds, default: 300000 (5 min)
}

Implementation:

• Default timeout: 5 minutes
• Allow override for known long operations (CI polling, builds)
• Kill process tree on timeout
• Return timeout error to caller

3. Async Message Processing (High Priority)

Current: Serial message processing - one message blocks all others

Proposed: Concurrent message handling with worker pool

┌──────────────┐
│ Message Queue│
└──────┬───────┘
       │
   ┌───▼────┬────────┬────────┐
   │Worker 1│Worker 2│Worker 3│  ← Process messages concurrently
   └────────┴────────┴────────┘

Benefits:

• One slow message doesn't block others
• Natural load balancing
• Better resource utilization
• Graceful degradation under load

Considerations:

• Message ordering (when required)
• Shared state management
• Resource limits (max concurrent)

4. Health Monitoring & Auto-Recovery (High Priority)

Add: Watchdog process that monitors control-agent health

Check every 30-60 seconds:

• Time since last message processed
• Queue depth
• Process responsiveness
• Active child processes

Auto-recovery when stuck:

1. Log state to incident file
2. Identify blocking child processes
3. Kill blocking processes
4. Verify recovery
5. Alert operator

Metrics to expose:

• Messages processed per minute
• Average processing time
• Current queue depth
• Worker count
• Time since last activity

5. Circuit Breaker for Operations (Medium Priority)

Pattern: Fail fast when operations consistently fail

Operation fails 3 times → 
  Circuit opens (skip attempts for 5 min) → 
  Retry after timeout → 
  Success → Circuit closes

Apply to:

• Worker spawns
• API calls
• External service calls

6. Graceful Degradation (Medium Priority)

Strategies when under load:

1. Immediate acknowledgment: Always respond within 5 seconds

   • "Got it! Working on this..."

2. Queue position updates:

   • "Request queued, position docs: add git workflow rules to AGENTS.md #3 of 7"

3. Auto-fallback responses:

   • After 2 min: "System under heavy load, will respond shortly"

4. Priority queue:

   • Some messages (admins, urgent) jump queue

7. Observability Improvements (Medium Priority)

Add status endpoints/tools:

• /health - Is control-agent responsive?
• /metrics - Queue depth, processing rate, worker count
• /status - Current state, last activity timestamp

Structured logging:

{
  "timestamp": "2026-02-27T04:08:01Z",
  "level": "warn",
  "component": "control-agent",
  "event": "spawn_timeout",
  "worker_id": "dev-agent-xyz",
  "duration_ms": 300000
}

8. Worker Lifecycle Tracking (Low Priority)

Track state transitions:

SPAWNING → READY → ASSIGNED → WORKING → REPORTING → COMPLETE/FAILED

Benefits:

• Identify slow spawns
• Detect stuck workers
• Optimize performance
• Better resource cleanup

Testing Recommendations

Load Testing

1. Spawn stress: Spawn 10 workers rapidly, verify no blocking
2. Message flood: 20 messages in 10 seconds, verify all processed
3. Long operations: Start 5-min operation, verify other work continues

Chaos Testing

1. Kill random worker processes
2. Block network for 5 minutes
3. Fill disk to 99%
4. Send malformed messages

Recovery Testing

1. Verify auto-recovery from blocked state
2. Verify graceful shutdown
3. Verify worker cleanup on crash

Success Criteria

• Control-agent never blocks for >30 seconds
• All incoming messages acknowledged within 5 seconds
• Messages processed concurrently when independent
• System auto-recovers from blocked states
• Health metrics visible and monitored
• P95 response time < 30 seconds
• Uptime > 99.9%

Additional Context

This issue was discovered during production operation when a worker spawn hung for 22 minutes, causing complete control-agent unresponsiveness. The worker itself functioned correctly once spawned, but the control-agent remained blocked waiting for the spawn command to return.

Manual intervention (killing blocking processes) immediately restored service, but highlighted the need for:

1. Asynchronous operations by default
2. Automatic health monitoring
3. Self-recovery mechanisms
4. Better visibility into system state

Related Work

• Consider similar patterns in other agents (sentry-agent, dev-agent)
• Review all bash tool usage for potential blocking operations
• Audit all external process spawning
• Review timeout usage across codebase

---

Priority: Critical
Effort: 2-3 weeks (all high-priority items)
Impact: Eliminates entire class of outage scenarios

[vltbaudbot]

Post-Mortem: Control-Agent Inbox Blocking (20+ minute outage)

Incident Summary

Date: 2026-02-27
Duration: 22 minutes (04:08 - 04:30 UTC)
Severity: Critical - Complete control-agent unresponsiveness
Root Cause: Synchronous worker spawn command blocking control-agent's main event loop

Timeline

• 04:08:01 - Control-agent executes pi session spawn command synchronously
• 04:08:01 - 04:30:50 - Command hangs/never completes, control-agent completely blocked
• 04:08 - 04:26 - 7 incoming messages queued by bridge, never reach control-agent
• 04:30:50 - Debug-agent identifies issue and kills blocking process
• 04:30:51 - Control-agent resumes, processes all queued messages within 30 seconds

Impact

User Experience

• 22 minutes of complete silence on all incoming requests
• 7 messages went unanswered during the outage
• Users received initial acknowledgments but no follow-up for 20+ minutes
• Loss of trust in system responsiveness

System Behavior

• Bridge continued receiving and forwarding messages (functioning correctly)
• Worker agent spawned successfully and worked independently (created PR, ran tests)
• Control-agent process remained alive but completely unresponsive to new input
• No automatic recovery mechanism triggered

Root Cause Analysis

Primary Cause: Synchronous Blocking I/O

The control-agent executed a worker spawn command using the bash tool:

pi session spawn \
  --name "dev-agent-..." \
  --cwd ~/workspace/worktrees/... \
  --skill dev-agent \
  --provider anthropic \
  --model claude-sonnet-4-20250514 2>&1 | tail -5

Problem: This command is executed synchronously - the control-agent waits for it to complete before processing any other messages.

Why it hung:

• The spawn process started successfully (PID 2078108)
• The spawned worker began working immediately
• However, the parent bash process (2078106) never returned
• The tail -5 in the pipeline kept running
• Control-agent's bash tool waited indefinitely for the command to complete

Contributing Factors

1. No timeout on bash tool execution - Command can run forever
2. Serial message processing - Single-threaded inbox processing
3. No health monitoring - No automatic detection of stuck state
4. No circuit breaker - No fallback when spawn takes too long
5. No visibility - Users had no indication system was blocked

Immediate Fixes Implemented

1. Operational Guidance Updated

Added to ~/.pi/agent/memory/operational.md:

• Non-blocking spawn pattern - Always background worker spawns
• Parallelization guidance - Process multiple tasks concurrently
• Status update protocol - Proactive updates at 2/5/10 minute intervals

2. Emergency Recovery

• Killed blocking processes manually
• Control-agent automatically resumed processing
• All queued messages handled successfully

Recommended Changes

High Priority (Critical)

1. Non-Blocking Worker Spawns

Problem: Synchronous spawn blocks control-agent for minutes
Solution: Always spawn in background with verification

# Spawn non-blocking
(pi session spawn \
  --name "dev-agent-..." \
  --cwd ~/path \
  --skill dev-agent \
  > /tmp/spawn-${TODO_ID}.log 2>&1 &)

# Wait briefly for socket to appear
sleep 3

# Verify and continue immediately
if [ -S ~/.pi/session-control/dev-agent-....sock ]; then
  # Send task to worker
  send_to_session --sessionName "dev-agent-..." --message "..."
fi
# Continue processing inbox - don't wait for worker!

Implementation:

• Update control-agent SKILL.md with non-blocking spawn examples
• Add warning in dev-agent spawn documentation
• Create helper script for safe spawning

2. Bash Tool Timeouts

Problem: Bash commands can run indefinitely
Solution: Default timeout on all bash tool invocations

// In bash tool implementation
const DEFAULT_TIMEOUT = 300; // 5 minutes
const timeout = options.timeout || DEFAULT_TIMEOUT;

Edge cases to handle:

• Long-running CI polling (gh pr checks --watch)
• Build commands
• Test suites
• Allow override for known long operations

3. Async Message Processing Architecture

Problem: Control-agent processes messages serially
Solution: Concurrent message handling with work queue

Proposal:

┌─────────────────┐
│  Slack Bridge   │
│   (receives)    │
└────────┬────────┘
         │
         v
┌─────────────────┐
│  Control Agent  │
│  ┌───────────┐  │
│  │ Message   │  │
│  │ Queue     │  │
│  └─────┬─────┘  │
│        │        │
│   ┌────v─────┐  │
│   │ Worker   │  │  ← Handles one message
│   │ Pool     │  │     Returns to queue
│   └──────────┘  │
└─────────────────┘

Benefits:

• Multiple messages processed concurrently
• One slow operation doesn't block others
• Natural load balancing
• Graceful degradation under load

4. Health Monitoring & Auto-Recovery

Problem: No detection of stuck/blocked state
Solution: Watchdog process with automatic recovery

Implementation:

# Watchdog extension (runs every 30 seconds)
- Check: Last message processed timestamp
- If > 5 minutes since last activity:
  - Check: Are there queued messages?
  - Check: Is process responsive? (can send internal message)
  - If unresponsive:
    - Log state to incident file
    - Kill blocking child processes
    - Attempt graceful recovery
    - Alert operator

Metrics to track:

• Messages processed per minute
• Average message processing time
• Queue depth
• Time since last activity
• Active worker count

Medium Priority (Important)

5. Circuit Breaker for Spawn Operations

Problem: Failed spawns retry multiple times, compounding delay
Solution: Circuit breaker pattern

Spawn attempt fails → 
  Circuit opens → 
  Skip spawns for 5 min → 
  Try again → 
  Success → Circuit closes

Parameters:

• Failure threshold: 3 consecutive failures
• Timeout period: 5 minutes
• Success threshold: 2 consecutive successes

6. Visibility & Observability

Problem: No external visibility into control-agent state
Solution: Status dashboard and metrics

Expose:

• /health endpoint: Is control-agent responsive?
• /metrics endpoint: Queue depth, processing rate, worker count
• /status endpoint: Current state, last activity, active tasks
• Dashboard widget: Real-time control-agent state for debug-agent

7. Graceful Degradation

Problem: System has no fallback when control-agent is blocked
Solution: Multiple degradation strategies

Strategies:

• Acknowledge immediately - Always send "Got it, working on this!" within 5 seconds
• Queue visible to bridge - Bridge can respond with "Request queued, position docs: add git workflow rules to AGENTS.md #3"
• Fallback response - After 2 minutes, bridge auto-responds: "System under heavy load, will respond shortly"
• Priority queue - Some messages (e.g., from admins) jump queue

8. Worker Lifecycle Management

Problem: No tracking of worker spawn → task → completion lifecycle
Solution: Structured worker management

States:

SPAWNING → READY → ASSIGNED → WORKING → REPORTING → COMPLETE/FAILED

Track:

• Spawn start/end time
• Socket availability time
• Task assignment time
• First progress report time
• Completion report time

Benefits:

• Identify slow spawns
• Detect stuck workers
• Optimize spawn performance
• Better resource management

Low Priority (Nice to Have)

9. Structured Logging

Current: Mixed stdout/jsonl logging
Proposed: Structured JSON logs with levels

{
  "timestamp": "2026-02-27T04:08:01Z",
  "level": "info",
  "component": "control-agent",
  "event": "worker_spawn_start",
  "worker_id": "dev-agent-abc123",
  "todo_id": "TODO-xyz",
  "metadata": {}
}

10. Message Deduplication at Bridge

Current: Deduplication cache in bridge (working well)
Enhancement: Persist cache across restarts

11. Retry with Exponential Backoff

For: Failed spawns, API errors, transient failures
Pattern: 1s, 2s, 4s, 8s, 16s, stop

12. Resource Limits

Track:

• Max concurrent workers (e.g., 5)
• Max spawn attempts per hour (e.g., 20)
• Max memory per worker
• Max queue depth before rejection

Testing Recommendations

Load Testing

1. Spawn stress test - Spawn 10 workers rapidly, verify no blocking
2. Message flood test - Send 20 messages in 10 seconds, verify all processed
3. Long operation test - Start 5-minute CI poll, verify other work continues
4. Spawn failure test - Force spawn failures, verify graceful handling

Chaos Testing

1. Kill random worker processes mid-task
2. Block Slack bridge for 5 minutes
3. Fill disk to 99%
4. Saturate network bandwidth
5. Send malformed messages

Recovery Testing

1. Verify automatic recovery from blocked state
2. Verify queue persistence across restarts
3. Verify worker cleanup on crash
4. Verify graceful shutdown

Metrics to Track Going Forward

Response Time

• P50: 50th percentile response time (should be < 5s)
• P95: 95th percentile response time (should be < 30s)
• P99: 99th percentile response time (should be < 2m)

Throughput

• Messages processed per minute
• Workers spawned per hour
• Tasks completed per hour

Reliability

• Uptime percentage (target: 99.9%)
• Mean time between failures (MTBF)
• Mean time to recovery (MTTR)

Resource Utilization

• Control-agent CPU usage
• Control-agent memory usage
• Worker count (current/max)
• Disk usage for logs/sessions

Lessons Learned

What Went Well

• Bridge continued operating correctly during outage
• Worker spawned successfully and completed work despite control-agent being blocked
• Session JSONL files provided complete audit trail
• Debug-agent had tools to diagnose and fix issue
• Recovery was immediate once blocking processes were killed

What Went Poorly

• No automatic detection of blocked state
• No visibility into system health for operators
• Users experienced complete silence for 20+ minutes
• No fallback or degradation strategy
• Serial processing amplified impact of single blocking operation

What We'll Do Differently

• Design for asynchrony - Never block on I/O operations
• Monitor everything - Track all state transitions and durations
• Fail fast - Use timeouts everywhere
• Degrade gracefully - Have fallback responses
• Make problems visible - Expose metrics and health checks

Action Items

#	Action	Priority	Owner	Status
1	Implement non-blocking worker spawns	Critical	-	Open
2	Add default timeouts to bash tool	Critical	-	Open
3	Design async message processing architecture	Critical	-	Open
4	Implement health monitoring watchdog	Critical	-	Open
5	Add circuit breaker for spawn operations	High	-	Open
6	Create status dashboard/metrics endpoint	High	-	Open
7	Implement graceful degradation strategies	High	-	Open
8	Build worker lifecycle tracking	Medium	-	Open
9	Add structured logging	Low	-	Open
10	Set up load testing suite	High	-	Open

Conclusion

This incident revealed a fundamental architectural issue: synchronous blocking operations in a single-threaded event loop cause complete system unavailability. The fix is well-understood (asynchronous operations with timeouts), but requires careful implementation across multiple components.

The immediate operational fixes prevent recurrence in the short term, but the recommended architectural changes are necessary for long-term reliability and scalability.

Estimated effort for all Critical/High priority fixes: 2-3 weeks development + testing.

---

Document prepared by: Debug-agent
Incident resolved: 2026-02-27 04:30 UTC
Recovery time: < 1 minute after intervention

[vltbaudbot]

Addressed the remaining actionable items from this issue in PR #198:

Implemented:

• ✅ Circuit breaker for agent_spawn — opens after 3 consecutive failures, 5min cooldown
• ✅ Worker lifecycle logging — all spawn events to worker-lifecycle.jsonl
• ✅ spawn_status tool — inspect circuit breaker state and recent events
• ✅ Heartbeat auto-recovery — bridge-down and orphaned dev-agents auto-fixed before prompting

Previously done:

• ✅ Non-blocking spawns via agent_spawn tool (extensions: add agent_spawn tool for bounded dev-agent startup #188)

Requires pi core changes (not addressable here):

• ⏳ Default bash tool timeouts
• ⏳ Async message processing / concurrent message handling