Skip to content

Conversation

@devin-ai-integration
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Jan 8, 2026

Summary

Adds a microphone button labeled "Ask Voice Agent" to the right of the Ask AI button (or Search button as fallback) in the docs header that enables real-time voice chat with an AI assistant. When clicked, the button connects to a WebSocket endpoint (wss://aaigentsv1.up.railway.app/ws/assemblyai_docs_agentt), streams microphone audio to the server, and plays back audio responses.

The implementation:

  • Creates a button with microphone icon and "Ask Voice Agent" text that appears next to the Ask AI button (if present) or falls back to appearing next to fern-search-button
  • Shows button state transitions: "Ask Voice Agent" → "Connecting..." (amber) → "Disconnect" (red) → back to "Ask Voice Agent"
  • Handles WebSocket connection for bidirectional audio streaming
  • Converts audio between Float32 (Web Audio API) and PCM16 (server format)
  • Uses a MutationObserver to ensure the button is added even if the Ask AI or search button loads dynamically
  • Only starts audio capture after receiving session.created from the server (scriptProcessor is not connected until then to prevent any buffering)
  • Uses scheduled audio playback with latency management (resets queue if more than 300ms backed up)
  • Displays a chatbox dropdown showing conversation transcripts (both interim and final messages)

Updates since last revision

  • Fixed button text visibility: "Connecting..." text now explicitly styled white to be visible on amber background
  • Improved chatbox styling: cleaner design with blue gradient header, modern message bubbles (user messages white with border, assistant messages blue)
  • Added recording notice: chatbox now shows "This session is recorded to provide the service." before conversation starts
  • Removed "Listening..." placeholder text per feedback

Review & Testing Checklist for Human

  • Verify button text visibility: Click button and confirm "Connecting..." text is visible (white text on amber background)
  • Verify button state transitions: Confirm button shows "Connecting..." (amber), then changes to "Disconnect" (red) after session starts, then back to "Ask Voice Agent" when stopped
  • Test chatbox appearance: Verify the chatbox has a clean blue gradient header and the recording notice appears initially
  • Verify recording notice disappears: Once you speak, confirm the notice is replaced by conversation messages
  • Test audio playback latency: Speak and verify the AI response plays back promptly without noticeable delay
  • Verify WebSocket URL: Confirm wss://aaigentsv1.up.railway.app/ws/assemblyai_docs_agentt (note: double 't') is the correct endpoint

Recommended test plan:

  1. Deploy to preview environment and verify button appears with "Ask Voice Agent" text
  2. Click button and verify it changes to "Connecting..." with visible white text on amber background
  3. Grant microphone permission and verify button changes to "Disconnect" (red color)
  4. Verify chatbox appears with blue header and recording notice text
  5. Speak a question and verify the notice disappears and your words appear in the chatbox
  6. Verify the AI responds with audio that plays back promptly (no significant delay)
  7. Click "Disconnect" and verify button returns to "Ask Voice Agent" and chatbox disappears
  8. Test in multiple browsers

Notes

  • The ScriptProcessorNode API used for audio processing is deprecated but still widely supported
  • Audio latency fix uses 300ms threshold and 50ms buffer - these values may need tuning based on real-world testing
  • Chatbox uses absolute positioning relative to button parent - verify it doesn't cause layout issues in different viewport sizes
  • Pre-existing lint errors in openapi.yml are unrelated to this change

Link to Devin run: https://app.devin.ai/sessions/54dfefda02354e80a13914c9e5e77b03
Requested by: Griffin Sharp (@gsharp-aai)

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link

github-actions bot commented Jan 8, 2026

…ket URL

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

…back

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

Co-Authored-By: Griffin Sharp <gsharp@assemblyai.com>
@github-actions
Copy link

github-actions bot commented Jan 9, 2026

@dan-ince-aai dan-ince-aai merged commit fdd48c0 into main Jan 9, 2026
2 of 4 checks passed
@dan-ince-aai dan-ince-aai deleted the devin/1767915167-voice-agent-button branch January 9, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants