Do video-call agents look? An open reproduction of VideoFDB

A good listener on a video call does a lot without taking the floor. They stay quiet when you pause to think, keep talking when you nod, stop when you raise a hand, and laugh when you laugh. VideoFDB (Mazumdar et al., 2026), a benchmark from NVIDIA and David AI for full-duplex audio-visual agents (agents that listen and speak at the same time), asks whether current video-call agents do any of this, and at the right moment.

NVIDIA released the clips (gated) and described the protocol, but not the evaluation code or a scorer. OpenVideoFDB is our open implementation of the whole pipeline, plus an open clip set for anyone without access to the NVIDIA data.

What the benchmark measures

Each clip is a short stretch of a real two-person conversation, around one of eleven dynamics: pause handling, gaze aversion while pausing, self-touching gestures, nonverbal and verbal interruption, facial emotion, nodding and smiling, laughter, emotion matching, verbal backchannels (“mhm”), and ordinary turn-taking. The agent hears the user’s side in real time and sees one video frame per second. What it says back is scored two ways:

• Timing. A deterministic rule per dynamic decides whether the agent should have spoken in the annotated window. TOR-Alignment (takeover-rate alignment) is the percentage of clips where it did the right thing; we also report the median latency in milliseconds.
• Quality. An LLM judge scores the reply on 1–5 rubrics: fluency, conversational flow and visual grounding for audio replies; fluency, dyadic affect and nonverbal-cue appropriateness for video replies. We ported the paper’s rubrics verbatim.

Settings follow the paper: AV2A is audio and video in, audio out; A2A is the same agent with the video withheld; AV2AV is video out, which is how the human reference is scored.

What we built

• A real-time harness that streams a clip’s audio and frames into a live agent and records its speech with word timestamps.
• The timing metrics and the judge, with a calibrated evidence payload for the judge.
• Adapters for Gemini Live and OpenAI Realtime, a human-reference player (the real partner’s side of the recording) and a random-partner baseline.
• Adapters for three hosted avatar products (Tavus CVI, Anam and HeyGen LiveAvatar).
• A loader for NVIDIA’s dataset, so the same code scores the real benchmark if you have access.
• An open clip set mined from Meta’s Seamless Interaction corpus, published as manifests on Hugging Face that re-cut the clips deterministically from the public recordings.

Results on the NVIDIA test split

All 226 rows of the test split, same clips, windows, captions and speech recognition as the paper, judged by gpt-4o. Quality columns are judge scores on a 1–5 scale; Timing is TOR-Alignment (% of clips) / median latency (ms). The last column is the paper’s overall score / TOR-Alignment / latency.

The human reference can only be computed on the 60 rows where NVIDIA ships the partner’s channel, and there it lands at 4.22 overall against the paper’s 4.20. Both API models land within 0.25 points of the paper overall.

The paper’s most striking finding reproduces: removing the video makes no difference. Each model’s AV2A and A2A scores differ by about 0.1 points with overlapping confidence intervals. The agents are not using the camera on these dynamics. Two differences from the paper are worth noting: our TOR-Alignment is 56–60 % against the paper’s 66–72 %, and our gpt-realtime latency is 1.1–1.4 s where the paper reports 4.4–5.4 s, which we trace to how speech onset is defined.

Avatars do not react while you talk

The generation part of the benchmark (121 rows) asks for video out. We ran three hosted avatar products, all cascaded pipelines: speech recognition, then a language model, then speech synthesis, then a lip-synced renderer.

None of them laughs, nods or smiles as a reaction inside the window. On all 30 laughter and nodding rows, Tavus and Anam score 0 on the nonverbal-cue rubric; HeyGen smiles once on each. That matches the paper’s observation that cascaded avatars cannot insert a nonverbal cue during the user’s turn: nothing happens on the face until the language model has something to say. HeyGen and Tavus are not separable overall (paired difference +0.12 points, 95 % confidence interval −0.12 to +0.35), and both score above Anam.

Two findings that hold on both data sources

We also ran the API models on 60 rows of our own Seamless-derived set. Two results hold there as well as on the NVIDIA data:

1. Current realtime models do not use the video channel on these dynamics. A2A scores are at least as high as AV2A almost everywhere.
2. No model produces a verbal backchannel (“mhm”, “right”) while the user holds the floor.

Both point at the same gap: today’s agents treat a conversation as alternating monologues. The listening behaviour that people do without thinking, such as nodding, holding back through a pause, or acknowledging without interrupting, is not there yet.

Try it

git clone https://github.com/OpenHRIai/OpenVideoFDB && cd OpenVideoFDB
uv sync
uv run pytest     # 160 tests, no network needed

The README covers scoring the NVIDIA split (needs access) and rebuilding the open clip set from the Hugging Face manifests. Every known deviation from the paper is listed in the repository’s docs/protocol.md.

@misc{openhri2026openvideofdb,
  title  = {Do video-call agents look? An open reproduction of VideoFDB},
  author = {Maxim Makatchev},
  year   = {2026},
  note   = {OpenHRI research blog},
  url    = {https://openhri.ai/research/open-videofdb/}
}