NVIDIA and Other Institutions Release Realtime-Bench: Frontier Large Models Fall Short on Real-Time Tasks
3 day ago / Read about 0 minute
Author:小编   

NVIDIA, in collaboration with UT-Austin and other institutions, has launched Realtime-Bench, a benchmark designed for real-time interactive tasks. The benchmark comprises 50 tasks spanning eight scenario types, with a focus on evaluating three model capabilities: real-time perception, time-adaptive reasoning, and action timing. It also introduces three environment clock-advancement protocols to isolate the effects of different factors. Using this benchmark, the researchers tested 12 frontier large models. The results reveal that, in scenarios where the environment advances according to actual inference latency, the best-performing model, GPT-6-Astra, achieves only a 44% task success rate—and under high reasoning intensity, the probability of all three runs succeeding drops to just 36%. Most models struggle to strike a balance between low latency and reasoning capability. The experiments further indicate that in real-time tasks, increasing reasoning intensity does not necessarily improve success rates; models must trade off between depth of thought and timing of action, and simply reducing inference latency is insufficient to address all real-time interaction challenges.

  • C114 Communication Network
  • Communication Home
7 X 24 Track global technological trends
Hot Topic