← Back to PRs

#14996: feat(matrix): pass asVoice flag through message tool and add waveform generation

by mauldus open 2026-02-12 21:07 View on GitHub →
channel: matrix size: S
The message tool's asVoice parameter was accepted by the schema but never passed through the Matrix plugin's action handler chain. This meant org.matrix.msc3245.voice was never set on outgoing audio events, causing Element clients to render voice messages as file attachments instead of inline voice bubbles. Changes: - Pass asVoice from action handler → tool-actions → messages → send - Add generateWaveform() using ffmpeg to extract PCM amplitude peaks - Include waveform data in org.matrix.msc1767.audio for Element's voice message visualization (progress bar with amplitude wave) Fixes voice messages sent via the message tool not rendering as native voice bubbles in Element (Desktop, iOS, Android). (AI Assisted, fully tested, fully understood the change) <!-- greptile_comment --> <h2>Greptile Overview</h2> <h3>Greptile Summary</h3> This PR threads an `asVoice`/`audioAsVoice` boolean through the Matrix message tool/action chain so outgoing audio uploads can be marked as voice messages (adds `org.matrix.msc3245.voice` and `org.matrix.msc1767.audio`). It also adds optional waveform generation for voice messages by calling `ffmpeg` to downmix/resample audio and compute 100 peak buckets (0–1024), attaching the waveform to `org.matrix.msc1767.audio.waveform` so Element can render the voice-message waveform UI. Changes are localized to the Matrix extension send pipeline (`actions.ts` → `tool-actions.ts` → `matrix/actions/messages.ts` → `matrix/send.ts` → `matrix/send/media.ts`) and don’t alter non-media message sending behavior. <h3>Confidence Score: 4/5</h3> - This PR is reasonably safe to merge and is scoped to Matrix voice-message metadata and optional waveform extraction. - I didn’t find any definite runtime/type errors introduced in the changed code after verifying the only suspicious pattern (`encoding: "buffer"`) is already used elsewhere in the repo. The ffmpeg path is best-effort and falls back cleanly when unavailable/failing, limiting blast radius. The main remaining risk is environment-specific ffmpeg behavior (stderr volume, input probing), but that isn’t provably broken from the diff alone. - extensions/matrix/src/matrix/send/media.ts (ffmpeg waveform generation path) <sub>Last reviewed commit: 35e597b</sub> <!-- greptile_other_comments_section --> <sub>(2/5) Greptile learns from your feedback when you react with thumbs up/down!</sub> <!-- /greptile_comment -->

Most Similar PRs