← Back to PRs

#16590: fix(web-fetch): use bot UA for markdown to enable Cloudflare LLM content negotiation

by Imccccc open 2026-02-14 22:23 View on GitHub →
agents size: XS
## Summary Use `OpenClaw/1.0 (AI Agent)` User-Agent when `extractMode='markdown'` is requested, instead of the default Chrome User-Agent. ## Why? Cloudflare's [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) feature relies on content negotiation. However, if the client sends a standard Browser User-Agent (like Chrome), Cloudflare serves HTML (for humans) even if `Accept: text/markdown` is present. By switching to a non-browser UA when markdown is explicitly requested, we correctly trigger Cloudflare's automatic HTML-to-Markdown conversion, resulting in cleaner, token-efficient content for agents. ## Testing Verified with `curl` that: - Chrome UA + Accept markdown -> HTML - OpenClaw UA + Accept markdown -> Markdown <!-- greptile_comment --> <h3>Greptile Summary</h3> This PR contains two separate changes: 1. **Web-fetch bot UA for Cloudflare Markdown content negotiation**: Switches from Chrome User-Agent to `OpenClaw/1.0 (AI Agent)` when `extractMode='markdown'` and the default UA is in use. This correctly triggers Cloudflare's [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) content negotiation, yielding cleaner markdown instead of HTML. Custom user agents from config are correctly preserved. 2. **Config-level `env` field for exec tool**: Adds an `env` field to `ExecToolConfig` allowing operators to inject environment variables into exec commands. The merge precedence is correct (`process.env < config.env < tool-call.env`), and security validation appropriately applies only to untrusted tool-call env vars. - The `nodeEnv` variable in `bash-tools.exec.ts` is now always a truthy object (even `{}`), making the `if (nodeEnv)` guard dead code and changing the serialized payload sent to remote nodes from `undefined` to `{}`. This warrants a quick check that the node receiver handles empty `env` objects the same as missing ones. <h3>Confidence Score: 4/5</h3> - This PR is safe to merge with one minor behavioral change in node env serialization worth verifying. - The web-fetch UA change is clean and well-scoped. The exec env merging follows existing patterns with correct security boundaries. The only concern is the subtle `nodeEnv` truthiness change that turns an `undefined` into `{}` in the node invoke payload, which could theoretically affect remote node handling. - `src/agents/bash-tools.exec.ts` — verify that the `nodeEnv` change from `undefined` to `{}` is handled correctly by remote nodes. <sub>Last reviewed commit: 8b5a2fc</sub> <!-- greptile_other_comments_section --> <!-- /greptile_comment -->

Most Similar PRs