Full-Duplex Voice Is Mostly a Problem of Knowing When to Stop

The demo is the easy part. Tap, talk, hear a reply. The thing that takes a month is that the reply has to stop when it is supposed to, the user has to be able to interrupt it, and the system must not confuse the assistant's own voice with the user's.

None of those are interesting-sounding problems. All of them are the product.

The four states

The whole system is a small state machine, and the UI has to show it honestly:

The state matters for more than looks. It tells the user whether their microphone is live, whether the model has heard them, and whether interrupting will work right now.

And it needs to be believable. A UI that says thinking for a fixed 400ms while the model actually takes 6 seconds is worse than no state display at all — it teaches the user not to trust the indicator.

The barge-in problem

The user talks over the assistant. Three things have to happen, and they have to happen in the right order:

1. Detect that the user is speaking while the assistant is speaking.
2. Stop the assistant's audio immediately — not at the end of the current sentence.
3. Discard the partial assistant response that was cut off, and start fresh from the user's new input.

Step 3 is the one people miss. If you stop the audio but keep the half-finished assistant turn in the conversation, the model now sees its own truncated response as if it had said it. It will apologize for being interrupted, or worse, continue from the truncated fragment as though it were a complete thought.

The correct behavior is: cut the audio, throw away the partial assistant message, and treat the user's interruption as a fresh turn.

Playback bleed

The microphone hears the speaker. This is the classic feedback loop.

The naive fix — ignore the mic while the speaker is active — breaks barge-in entirely, because barge-in requires hearing the user during playback.

What actually works is echo cancellation: cancel the known speaker signal from the microphone input, then detect speech on what remains.

There is a subtler version of the same problem. The assistant's own voice, partially echoed and partially delayed, can land in the transcript as if the user said it. Then the assistant responds to its own words. The symptom is strange and the cause is invisible.

Our test asserts that playback bleed does not trip the interrupt detector, and there is a specific test for the case where the agent was audibly speaking and got cut off — we call that state playback, and it is distinct from a genuine user interruption.

Latency, and the honest number

The pipeline is: capture → endpoint → transcribe → answer → synthesize → play.

Each step adds delay. The user does not perceive "correct," they perceive "instant." There is a threshold, and past it the interaction stops feeling like a conversation and starts feeling like a form submission.

The part that surprises people is that latency is not only compute time. It is also the time before you start speaking. If the endpoint detector waits 800ms to be sure the user is done talking, the whole round trip is 800ms longer before the assistant even begins to respond.

We found this the hard way. The fix was endpointing early and being willing to be wrong — start processing on a short pause, and if the user keeps talking, discard and restart. Discarding is cheap. Waiting is not.

The long-reply case

A model that takes 30 seconds to think, then produces a 45-second response, breaks the illusion completely. The user is left staring at a thinking indicator for half a minute with no idea whether the system heard them.

Two mitigations:

1. Keep it interruptible throughout. The thinking state is not a lockout. If the user says something, the model should stop and respond to that instead.
2. Stream the response. Synthesize and play in chunks rather than waiting for the whole reply. The user starts hearing an answer in about a second, and the remaining latency disappears behind speech.

The part that sounds like a joke but isn't

Our test suite has a test that synthesizes real audio with Piper and asserts that the server transcribes it correctly. It sounds like a party trick.

It is the only test that catches the whole class of "the audio path is subtly wrong" bugs: sample rate mismatches, resampling artifacts, clipping, a channel count that is wrong in a way that still produces audible-but-garbled speech. All of those pass every unit test you write against a mock. All of them fail this one.

*The implementation is ai_agent_chat — MIT licensed, Python and FastAPI, running entirely on your own hardware over a VPN.*