Turn Detection and Barge-In for Production Voice Agents in October 2026: Endpointing, Interruptions, and Keeping Transcripts Honest

Voice agents feel broken for one reason more than any other: bad turn-taking. The model can be smart, the voice can sound natural, and users will still hang up if the agent talks over them, waits awkwardly after they finish, or keeps reading a long answer after they say "wait, stop." This post walks through the practical pieces of turn detection and barge-in handling for a production voice agent, whether you run a cascaded pipeline (speech-to-text, LLM, text-to-speech) or a speech-native realtime model.

A call center agent wearing a headset. Production voice agents have to handle the same turn-taking a human agent does without thinking. Image: Call Center Agent.jpg by FiveOne51, CC BY-SA 3.0, via Wikimedia Commons.

The three problems you are actually solving

Turn-taking sounds like one feature, but in practice it splits into three separate jobs:

End-of-turn detection (endpointing): deciding when the user has finished speaking so the agent can respond.
Barge-in: detecting that the user started talking while the agent is speaking, and stopping playback fast.
State repair: making sure the conversation history reflects what the user actually heard, not what the model generated.

Teams usually get the first one roughly right, the second one half right, and forget the third one entirely. The third one is where the subtle bugs live.

End-of-turn detection: silence is a weak signal

The simplest endpointer is a voice activity detector (VAD) plus a silence timer: once the VAD reports no speech for some number of milliseconds, the turn is over. Open-source VADs such as Silero VAD and the WebRTC VAD are common starting points, and most realtime speech APIs expose a server-side VAD with a configurable silence duration.

The trouble is that a single silence threshold forces a bad tradeoff:

Too short and the agent jumps in while the user pauses to think, reads a number off a card, or says "my account number is... hold on."
Too long and every response starts with dead air, which users read as the agent being slow or confused.

There is no universal correct value. Pick a starting point, then tune it against recordings of your real users, because a support line where people read order numbers behaves very differently from a quick-question kiosk.

Make the threshold context-aware

A better approach combines the silence timer with what the user said. A few practical rules go a long way:

Lengthen the wait after incomplete phrases. If the partial transcript ends with "and", "so", "um", "my number is", or a trailing digit sequence that is shorter than expected, extend the silence window.
Shorten it after complete questions. A transcript that ends in a clear question or a short confirmation ("yes", "that's right") can be endpointed sooner.
Use the agent's own last question. If the agent just asked for a 16-digit card number, do not endpoint after eight digits just because the user paused.

Some stacks go further and run a small semantic turn-detection model on the partial transcript to predict whether the user is done. Several realtime voice frameworks and APIs now offer this as an option alongside plain VAD. If you use one, keep the silence timer as a fallback ceiling so a model mistake can never leave the line hanging forever.

Speculate, but do not commit

You can hide endpointing latency by starting LLM generation on the partial transcript before you are sure the turn is over. The rule is simple: generate speculatively, but do not play audio or run tools with side effects until the turn is confirmed. If the user keeps talking, cancel the speculative request and start again with the longer transcript. This trades some extra tokens for a noticeably snappier agent.

Barge-in: stopping fast is the whole feature

When the user starts talking over the agent, three things need to happen, in this order:

Stop audio playback on the client immediately. Do not wait for the server to tell you. If the client has a local VAD, it should duck or stop the agent's audio on its own the moment speech is detected.
Cancel upstream work. Abort the in-flight LLM generation and the text-to-speech stream so you stop paying for tokens and audio nobody will hear.
Flush buffers. Audio that was already synthesized and queued in a jitter buffer or network buffer must be discarded, or the agent will "finish its sentence" a second later.

Buffer flushing is the most commonly missed step. A server can cancel generation perfectly and the user still hears another half-second of speech because the client had already received it.

Echo is not barge-in

The biggest source of false barge-ins is the agent hearing itself. On speakerphones, laptops, and smart speakers, the agent's own voice leaks back into the microphone and the VAD reports "user speech." Defenses, roughly in order of importance:

Enable acoustic echo cancellation on the client. Browsers expose it through the echoCancellation audio constraint in getUserMedia, and mobile platforms have equivalent voice-processing modes.
Require a minimum duration of detected speech (for example a few hundred milliseconds) before treating it as barge-in, so a cough or a door slam does not cut the agent off.
Optionally require that the speech produces at least one recognized word before interrupting, at the cost of slightly slower interruptions.

Not every interruption means "stop"

Users make small listening noises such as "mm-hm", "yeah", and "okay" while the agent talks. Treating these as full interruptions makes the agent stop mid-sentence for no reason. A common pattern is to pause playback on any detected speech, then look at the transcript: if it is only a short backchannel phrase, resume; if it contains real content, cancel and handle it as a new turn. Keep the backchannel list short and conservative, since "no" and "wait" must always interrupt.

State repair: keep the transcript honest

Here is the bug that bites almost every team. The LLM generated a 120-word answer, text-to-speech started reading it, and the user interrupted after 20 words. If your conversation history stores the full 120-word answer, the model now believes the user heard everything. On the next turn it may say "as I mentioned" about something the user never heard, or skip a step the user needed.

The fix is to truncate the assistant message in history to what was actually played:

Track playback position on the client, in milliseconds of audio actually rendered to the speaker.
When barge-in happens, send that position back to the server.
Map the position back to text using word-level timestamps from your TTS provider, or an estimate based on audio duration if timestamps are not available.
Replace the stored assistant message with the played prefix, optionally marked as interrupted.

Some realtime speech APIs support this directly with a truncate operation on the last assistant item, which takes the item ID and the audio end time. Use it if your provider has it. If you run a cascaded pipeline, store something like this:

{
  "role": "assistant",
  "content": "Sure. To reset your router, first unplug it from power and wait",
  "metadata": {
    "interrupted": true,
    "played_ms": 4200,
    "generated_chars": 612
  }
}

A short note in the system prompt helps the model use this correctly, for example: "If your previous message is marked interrupted, the user only heard the text shown. Do not assume they heard anything after it."

Tool calls and interruptions

Be careful when an interruption lands while a tool call is running. Read-only tools (looking up an order status) can simply be cancelled or allowed to finish and be discarded. Tools with side effects (issuing a refund, booking an appointment) should not be silently abandoned: either let them finish and tell the user what happened, or design them to be idempotent so the model can safely retry after the user clarifies. Never let a barge-in leave you unsure whether a write happened.

Measuring turn-taking quality

You cannot tune what you do not measure. Log these per turn and review them weekly against real recordings:

Response latency: time from the user's last word to the first agent audio actually played on the device, not just the first byte from the server.
False endpoint rate: turns where the user kept talking within a second or two after the agent started, a strong sign the agent jumped in too early.
False barge-in rate: interruptions where the transcript was empty, a backchannel, or clearly the agent's own echo.
Interruption-to-silence time: how long the agent keeps talking after the user starts. This is where buffer-flush bugs show up.
Repeat-request rate: how often users say "what?" or "can you repeat that", which often points to truncation or state-repair bugs.

Listen to a sample of calls in each bucket. Metrics tell you where the problem is; recordings tell you why.

A practical rollout checklist

Start with VAD plus a silence timer, tuned on real recordings from your use case.
Add transcript-aware rules to extend the wait after incomplete phrases and expected long inputs like numbers.
Implement client-side playback stop and buffer flush on detected speech, with echo cancellation enabled.
Add a minimum speech duration and a short backchannel list to cut false interruptions.
Truncate interrupted assistant messages to the played prefix before the next model call.
Make side-effecting tools safe under interruption.
Instrument latency, false endpoints, false barge-ins, and interruption-to-silence time, and review them regularly.

The takeaway

Good voice agents are not defined by the model alone. They are defined by when they speak, how quickly they stop, and whether they remember the conversation the way the user experienced it. Get endpointing context-aware, make barge-in stop the audio on the device instantly, and keep the transcript honest after every interruption. Those three changes do more for perceived quality than swapping in a bigger model.

Search This Blog

Technology On the Net