Listening through the pause: streaming end-of-turn detection

A voice agent has to decide, many times a minute, whether the person it is talking to has finished. Answer too early and it talks over someone who was only pausing to think. Answer too late and every reply comes after an awkward gap. This decision, end-of-turn detection, sits in front of everything else the agent does.

Pipecat’s Smart Turn v3.2 is a strong open model for it: a small Whisper-based audio classifier that looks at the last 8 seconds of speech and says whether the utterance sounds complete. smart-turn-streaming keeps its encoder frozen and changes how and when it is asked.

One question is not enough

The stock setup asks the model exactly once: 0.2 seconds after voice activity detection (VAD) reports that the user stopped. If the model says “complete”, the turn ends; if not, the agent waits out a 3-second timeout. On real conversation, from the AMI Meeting Corpus, that single query fires on 42 % of mid-turn pauses, and still leaves about a quarter of real turn ends to the timeout.

The problem is that the model never learns anything from the silence itself. A sentence that sounds unfinished at 0.2 s of silence is much more likely to be finished at 1.2 s, and a model that only ever sees the moment the speech stopped cannot use that.

Scoring the silence

The streaming detector scores the conversation every 100 ms. Small new heads sit on top of the frozen encoder: a gated recurrent unit (GRU) reads the last second of log-mel audio features, and its output is combined with the encoder’s pooled vector and the current length of the silence to give $p_\text{end}(\tau)$, the probability that the turn is over after $\tau$ seconds of silence.

The heads were trained on windows that end 0 to 2.5 s after the end of speech, from Pipecat’s smart-turn-data-v3.2 (padded with each clip’s own room tone), plus 43 thousand real conversational pauses from AMI.

Decisions are taken at three points of silence, $\tau \in \{0.2, 0.5, 1.2\}$ seconds, against a threshold table fitted to a false-alarm budget: for example, far15 allows at most 15 % of mid-turn pauses to be cut off.

Starting the reply early

Pipecat can speculate: start generating the language model’s reply before the turn is committed and hold it until the turn ends. The detector drives this with two tables:

• eager, when $p_\text{end}(\tau) \ge \theta_\text{eager}(\tau)$: start the language model, but hold its reply;
• commit, when $p_\text{end}(\tau) \ge \theta_\text{commit}(\tau)$: end the turn and release it;
• cancel, if the user speaks again: discard the held reply.

$\theta_\text{commit}$ is fitted to a budget of turn splits; $\theta_\text{eager}$ to a budget of generations wasted on pauses. With a language-model latency $L$, the user hears the agent at

Results

Offline on the AMI test split (511 turn ends and 4,304 mid-turn pauses), the new heads separate turn ends from pauses much better than the frozen model, and the gap widens as the silence grows. AUC is the area under the ROC curve (receiver operating characteristic): 0.5 is chance, 1.0 is perfect.

End to end in Pipecat 1.14, on 80 turn ends and 80 mid-turn pauses from AMI replayed in real time through Silero VAD, with 0.8 s of language-model latency. The first three columns are percentages of the 80 pauses; response time is in seconds.

The one-stage rule trades speed for safety: turn splits drop from 42.5 % to 5 %, but the median reply comes later. The two-stage rule keeps the 5 % split rate and wins back most of the speed, at the cost of some language-model calls spent on pauses that turn out not to be turn ends.

On Pipecat’s own smart-turn-data-v3.2 test set, which contains no pauses, the heads match the frozen model (AUC 0.976 against 0.976 at the end of speech), so nothing is lost on the original task.

Cost

Everything runs on CPU with onnxruntime, 32 MB of model files in all. Measured on an Apple M4 with one thread: 0.02 ms per 20 ms audio frame for the features, 0.7 ms per head update every 100 ms, and 22 ms per encoder pass, which runs on a worker thread every 320 ms during speech and at fixed points in the silence.

Limitations

• The conversational evaluation is English only (AMI meetings). The heads stay within noise of the frozen model on isolated utterances in all 23 languages of Smart Turn’s data, but whether the pause thresholds transfer to other languages’ conversational timing is not measured.
• Meetings set a ceiling: below an 18 % false-alarm budget, about a quarter of turn ends still wait for the backstop, for every model.
• Long pauses, 1.5 s or more, remain the weak spot: 50 to 60 % false alarms at the 10 to 15 % budgets.

Use it

pip install "smart-turn-streaming[pipecat] @ git+https://github.com/OpenHRIai/smart-turn-streaming"

from eot.hub import model_paths
from eot.pipecat_analyzer import StreamingEOTAnalyzer, StreamingEOTParams

encoder_onnx, heads_onnx = model_paths()   # ~32 MB from huggingface.co/OpenHRI
analyzer = StreamingEOTAnalyzer(
    encoder_onnx=encoder_onnx,
    heads_onnx=heads_onnx,
    params=StreamingEOTParams.from_rule("far15", eager="far15_gen30"),
)

The README shows the full Pipecat wiring and standalone use; the models are on Hugging Face.

@misc{openhri2026smartturnstreami,
  title  = {Listening through the pause: streaming end-of-turn detection},
  author = {OpenHRI},
  year   = {2026},
  note   = {OpenHRI research blog},
  url    = {https://openhri.ai/research/smart-turn-streaming/}
}