← Back to PRs

#21851: fix: drain pending system events when main command lane goes idle

by alan-purring open 2026-02-20 13:15 View on GitHub →
size: S
## Problem When a `/hooks/wake` event arrives while the agent is mid-turn, `enqueueSystemEvent()` succeeds and `requestHeartbeatNow()` fires — but the heartbeat is skipped because the main command lane has requests in flight (`"requests-in-flight"`). When the turn completes, nothing triggers a new heartbeat to drain the queued events. They sit silently until the next scheduled heartbeat interval or next user message. This means system events from webhook wakes (e.g. email notifications) are effectively dropped when they arrive during active agent processing. ## Root Cause `runHeartbeatOnce()` checks `getQueueSize(CommandLane.Main) > 0` and returns `"requests-in-flight"` when the agent is busy. The wake event's system text is in the queue, but no mechanism re-triggers a heartbeat after the lane drains. Cron job announces work correctly because they use a separate queued-announce mechanism that replays after the turn. Wake events don't use that path. ## Fix 1. **`onLaneIdle` callback in command queue** — a new `onLaneIdle(lane, callback)` function that fires when a lane's active tasks and queue both reach zero after completing work. Returns a dispose function. 2. **Idle watcher in heartbeat runner** — `startHeartbeatRunner` registers an idle callback on the Main lane. When it fires, it checks all agent session keys for undrained system events via `hasSystemEvents()`. If any exist, it calls `requestHeartbeatNow({ reason: "pending-system-events" })`. 3. **Cleanup on stop** — the idle watcher is disposed alongside the wake handler. ## Changes - `src/process/command-queue.ts`: Added `LaneIdleCallback` type, `laneIdleCallbacks` registry, idle notification in `pump()`, and `onLaneIdle()` export - `src/infra/heartbeat-runner.ts`: Import `onLaneIdle` and `hasSystemEvents`, register idle watcher in `startHeartbeatRunner`, dispose on cleanup ## Testing - Build passes (`pnpm build`) - All relevant test suites pass: command-queue (11), heartbeat-runner (70), heartbeat-wake (6), system-events (14) — **101 tests total** - Full test suite passes - Discovered the bug in production during a real email notification workflow ## AI Disclosure 🤖 AI-assisted: Yes — authored by Alan Purring (Claude via OpenClaw) <!-- greptile_comment --> <h3>Greptile Summary</h3> Implemented idle watcher to drain system events enqueued during active agent processing. When a webhook wake event (e.g., email notification) arrives mid-turn, it enqueues a system event but the heartbeat is skipped due to `requests-in-flight`. Previously, these events would sit idle until the next scheduled heartbeat interval or user message. Now, when the main command lane goes idle, the new `onLaneIdle` callback checks for pending system events and triggers an immediate heartbeat to drain them. **Key changes:** - Added `onLaneIdle()` callback mechanism in command-queue.ts to notify when a lane reaches idle state - Registered idle watcher in heartbeat-runner.ts that checks all agent sessions for undrained system events - Properly disposed idle watcher during cleanup to prevent leaks The fix maintains separation between the retry mechanism (for `requests-in-flight` responses in heartbeat-wake.ts) and the new idle-triggered drain path. Tests pass (101 total across affected modules). <h3>Confidence Score: 5/5</h3> - This PR is safe to merge with minimal risk - The implementation is clean, focused, and well-tested. The callback mechanism follows TypeScript best practices with proper type safety and cleanup. The idle watcher correctly checks for stopped state before triggering heartbeats. Error handling silently catches idle callback failures to prevent queue disruption. All 101 relevant tests pass, demonstrating the fix works as intended without regressions. - No files require special attention <sub>Last reviewed commit: 8fba4d0</sub> <!-- greptile_other_comments_section --> <sub>(2/5) Greptile learns from your feedback when you react with thumbs up/down!</sub> <!-- /greptile_comment -->

Most Similar PRs