#22201: feature(context): extend plugin system to support custom context management

[Discussion thread](https://github.com/openclaw/openclaw/discussions/22251) ## Summary - **Problem**: OpenClaw's context management (compaction, assembly, etc.) is hardcoded in core, making it impossible for plugins to provide alternative context strategies. This hasn't been a problem for very long, because other context management strategies haven't existed. - **Why it matters**: The [Lossless Context Management paper](https://papers.voltropy.com/LCM) from Ehrlich and Blackman demonstrated a promising alternative direction for context management. I have implemented that approach for OpenClaw (with a number of improvements), have been running it for the last week, and to say it works well would be an understatement. That work is in the [lossless-claw](https://github.com/Martian-Engineering/lossless-claw) repository. - **What changed**: Added a ContextEngine plugin interface with full lifecycle hooks, a slot-based registry, legacy pass-through wrapper, and wired it into the agent run loop, /compact command, and subagent registry. Added scoped subagent methods and gateway request scope for plugin runtimes, as an important aspect of LCM is the ability to spawn subagents for deep context traversal. - **What did NOT change**: Zero changes to existing compaction logic. The LegacyContextEngine wraps existing behavior identically. No new dependencies. No changes to message format, API contracts, or user-facing config (beyond an optional contextEngine slot). ## Change Type - [x] Feature - [x] Refactor ## Scope - [x] Gateway / orchestration - [x] Skills / tool execution (plugin SDK) - [x] Memory / storage - [x] API / contracts ## Linked Issue/PR Makes sense in the context of the [lossless-claw](https://github.com/Martian-Engineering/lossless-claw) repository, which is the first alternative context management engine. ## User-visible / Behavior Changes - New optional `contextEngine` config slot in agent plugin configuration. If unset, `LegacyContextEngine` is used automatically — zero behavior change for existing users. - Plugins can now register context engines via api.registerContextEngine(id, factory). ## Security Impact (required) - New permissions/capabilities? **No** — context engines have the same access scope as existing compaction code - Secrets/tokens handling changed? **No** - New/changed network calls? **No** - Command/tool execution surface changed? **No** — plugin SDK gains `registerContextEngine` but this is an extension point, not a new attack surface - Data access scope changed? **No** — context engines access the same message data that compaction already accesses - Scoped subagent interface limits plugin runtime to `run/waitForRun/deleteSession/getSession` (no full gateway access) ## Repro + Verification ### Environment - OS: macOS 14 (arm64) - Runtime: Node 22.17.0 - Model/provider: N/A ### Steps 1. `git checkout feature/context-engine && pnpm install && pnpm build` 2. Run without any context engine plugin configured → should behave identically to main 3. Install `lossless-claw` plugin: `pnpm openclaw plugins install @martian-engineering/lossless-claw` 4. Configure `contextEngine: "lossless-claw"` in `plugins.slots` 5. Send first message -> `lossless-claw` will begin incremental compaction of past messages to build the summary DAG ### Expected - Without plugin: identical behavior to main branch - With plugin: context engine lifecycle hooks called at each phase ### Actual Both scenarios verified across multiple openclaw installations by different users. ## Evidence Attach at least one: - [x] 334-line test suite (`context-engine.test.ts`) covering interface contract, registry resolution, legacy wrapper, and lifecycle wiring - [x] `server-plugins.test.ts` extended with 159 lines for plugin registration and scoped runtime - [x] `gateway-request-scope.test.ts` for scoped subagent method isolation - [x] Full test suite: 774/776 passing (2 pre-existing failures on upstream main, verified independently) ## Human Verification (required) What you personally verified (not just CI), and how: - Verified: Running OpenClaw gateway with no context engine plugin — identical behavior to main - Verified: Running with lossless-claw plugin — full lifecycle (bootstrap, assemble, afterTurn, ingest, compact) working end-to-end - Verified: Subagent lifecycle notifications fire correctly on delete/complete/release/sweep - Verified: /compact routes through context engine when present - Edge cases: Legacy fallback when plugin fails to load, slot resolution with no registered engines, multiple engines registered (first match wins) - Not verified: Windows/Linux (macOS only) ## Compatibility / Migration - Backward compatible? **Yes** - Config/env changes? **Yes** — optional `contextEngine` slot added to plugin config schema - Migration needed? **No** ## Failure Recovery (if this breaks) - How to disable/revert this change quickly: delete `contextEngine: "lossless-claw"` entry from config - Files/config to restore: delete `contextEngine: "lossless-claw"` entry from config ## Risks and Mitigations - **Risk**: Context engine plugin crashes during assemble() → LLM gets no/wrong context - **Mitigation**: Legacy fallback wraps existing proven code path; plugin errors are caught and logged. If a plugin fails to bootstrap, gateway falls back to legacy. - **Mitigation**: All messages are still written to JSONL session file even with new context engines like `lossless-claw`. No data is ever lost. • **Risk**: **ownsCompaction** flag disables Pi compaction but plugin doesn't actually compact → unbounded context growth • **Mitigation**: Overflow guard in `attempt.ts` still triggers `contextEngine.compact()` on token budget exceeded, regardless of `ownsCompaction` flag. The flag only suppresses the automatic Pi compaction, not the overflow safety net. ## Use of AI - AI was used to write this code - It's highly tested, both via automated testing and actual usage <!-- greptile_comment --> <h3>Greptile Summary</h3> Introduced a `ContextEngine` plugin interface to enable alternative context management strategies beyond OpenClaw's hardcoded compaction logic. The implementation wraps existing behavior in a `LegacyContextEngine` for backward compatibility, adds full lifecycle hooks (`bootstrap`, `ingest`, `assemble`, `compact`, `afterTurn`, `prepareSubagentSpawn`, `onSubagentEnded`), and wires it into the agent run loop, `/compact` command, and subagent registry. - Added `ContextEngine` interface with `AssembleResult`, `CompactResult`, `IngestResult`, `BootstrapResult` types - Created `LegacyContextEngine` wrapper that delegates to existing `compactEmbeddedPiSessionDirect` - Added slot-based registry with `resolveContextEngine()` supporting config-driven selection - Wired `contextEngine.bootstrap()` into run attempt initialization when session file exists - Wired `contextEngine.assemble()` into context assembly pipeline after sanitize/validate/limit - Wired `contextEngine.afterTurn()` or fallback `ingest()`/`ingestBatch()` after turn completion - Routed overflow compaction through `contextEngine.compact()` with legacy params bridge - Added `ownsCompaction` flag with `applyPiAutoCompactionGuard()` to disable Pi's internal compaction - Exposed `api.registerContextEngine()` in plugin SDK - Added scoped subagent runtime (`run`, `waitForRun`, `getSessionMessages`, `deleteSession`) using `AsyncLocalStorage` for gateway request scope - Added `sessions.get` gateway method to retrieve session messages - Wired subagent lifecycle notifications (`deleted`, `completed`, `swept`, `released`) to `contextEngine.onSubagentEnded()` - Added 334-line test suite covering interface contract, registry, legacy wrapper, and lifecycle wiring <h3>Confidence Score: 4/5</h3> - This PR is safe to merge with low risk - the implementation is well-tested, properly scoped, and maintains full backward compatibility through the legacy wrapper. - Score reflects thorough design (clean interface, proper lifecycle hooks, comprehensive error handling), extensive test coverage (334-line context engine test suite + 159 lines plugin tests), zero breaking changes (legacy wrapper preserves existing behavior), and careful integration (bootstrap only on existing sessions, graceful fallbacks on errors). The scoped subagent runtime using `AsyncLocalStorage` is an appropriate pattern, and the `ownsCompaction` flag provides safe opt-out of Pi's auto-compaction. Minor deduction for potential edge cases around context engine errors during critical paths (bootstrap/assemble) being logged but not blocking execution. - No files require special attention - the implementation is clean and well-tested throughout _{Last reviewed commit: 5c491ac} <!-- greptile_other_comments_section --> <!-- /greptile_comment -->