#16590: fix(web-fetch): use bot UA for markdown to enable Cloudflare LLM content negotiation
agents
size: XS
## Summary
Use `OpenClaw/1.0 (AI Agent)` User-Agent when `extractMode='markdown'` is requested, instead of the default Chrome User-Agent.
## Why?
Cloudflare's [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) feature relies on content negotiation. However, if the client sends a standard Browser User-Agent (like Chrome), Cloudflare serves HTML (for humans) even if `Accept: text/markdown` is present.
By switching to a non-browser UA when markdown is explicitly requested, we correctly trigger Cloudflare's automatic HTML-to-Markdown conversion, resulting in cleaner, token-efficient content for agents.
## Testing
Verified with `curl` that:
- Chrome UA + Accept markdown -> HTML
- OpenClaw UA + Accept markdown -> Markdown
<!-- greptile_comment -->
<h3>Greptile Summary</h3>
This PR contains two separate changes:
1. **Web-fetch bot UA for Cloudflare Markdown content negotiation**: Switches from Chrome User-Agent to `OpenClaw/1.0 (AI Agent)` when `extractMode='markdown'` and the default UA is in use. This correctly triggers Cloudflare's [Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/) content negotiation, yielding cleaner markdown instead of HTML. Custom user agents from config are correctly preserved.
2. **Config-level `env` field for exec tool**: Adds an `env` field to `ExecToolConfig` allowing operators to inject environment variables into exec commands. The merge precedence is correct (`process.env < config.env < tool-call.env`), and security validation appropriately applies only to untrusted tool-call env vars.
- The `nodeEnv` variable in `bash-tools.exec.ts` is now always a truthy object (even `{}`), making the `if (nodeEnv)` guard dead code and changing the serialized payload sent to remote nodes from `undefined` to `{}`. This warrants a quick check that the node receiver handles empty `env` objects the same as missing ones.
<h3>Confidence Score: 4/5</h3>
- This PR is safe to merge with one minor behavioral change in node env serialization worth verifying.
- The web-fetch UA change is clean and well-scoped. The exec env merging follows existing patterns with correct security boundaries. The only concern is the subtle `nodeEnv` truthiness change that turns an `undefined` into `{}` in the node invoke payload, which could theoretically affect remote node handling.
- `src/agents/bash-tools.exec.ts` — verify that the `nodeEnv` change from `undefined` to `{}` is handled correctly by remote nodes.
<sub>Last reviewed commit: 8b5a2fc</sub>
<!-- greptile_other_comments_section -->
<!-- /greptile_comment -->
Most Similar PRs
#15251: feat(web-fetch): send Accept: text/markdown header for Cloudflare M...
by wujieli0207 · 2026-02-13
86.8%
#15414: feat(web-fetch): add Accept: text/markdown header for Cloudflare Ma...
by aldoeliacim · 2026-02-13
84.7%
#15530: docs(web_fetch): document markdown-first Accept header and cf-markd...
by novavale · 2026-02-13
79.1%
#15242: feat(web-fetch): Add Accept header for Cloudflare Markdown for Agen...
by drkraft · 2026-02-13
77.3%
#20423: fix(web-fetch): cap htmlToMarkdown input size to prevent catastroph...
by Limitless2023 · 2026-02-18
75.9%
#9710: fix(ui): prevent CPU spike when opening large tool outputs (#9700)
by divol89 · 2026-02-05
74.9%
#9200: Fix: Strip dangerous env vars from baseEnv in host execution
by vishaltandale00 · 2026-02-05
73.4%
#19042: Security: add URL allowlist for web_search and web_fetch
by smartprogrammer93 · 2026-02-17
73.3%
#22644: feat(web-fetch): add allowPrivateNetwork config for web_fetch
by qingxuecc · 2026-02-21
73.3%
#14944: fix(browser): prefer openclaw profile in headless/noSandbox environ...
by BenediktSchackenberg · 2026-02-12
73.1%