Photo credit: www.sciencedaily.com
Emerging advancements in streaming technology have the potential to enhance user experiences in virtual reality (VR) and augmented reality (AR) environments, as outlined in recent findings from the NYU Tandon School of Engineering.
The study, which was shared during the 16th ACM Multimedia Systems Conference on April 1, 2025, introduces a novel method for predicting visible content within immersive 3D spaces. This technique could result in bandwidth reductions of up to seven times while preserving high visual quality.
This innovative technology is currently being utilized in a National Science Foundation-funded initiative at NYU Tandon, aimed at streamlining point cloud video for dance education. This development will enable 3D dance instruction to be accessible on standard devices with significantly lower bandwidth needs.
“One of the fundamental challenges in streaming immersive content has always been the extensive data requirements,” noted Yong Liu, a professor in the Electrical and Computer Engineering Department at NYU Tandon and a member of both the Center for Advanced Technology in Telecommunications and NYU WIRELESS. Liu led the research team responsible for these findings. “Conventional video streaming transmits everything visible in a frame, but our new approach mimics the way our eyes function, focusing only on the elements that are in a user’s line of sight.”
This new methodology tackles the challenge of “Field-of-View (FoV)” in immersive applications. Current AR and VR systems necessitate high data bandwidth; for instance, a point cloud video featuring 1 million points per frame can require over 120 megabits per second, nearly tenfold the amount used by standard high-definition video.
Unlike traditional methods that forecast where a user will direct their attention and then determine visibility, this cutting-edge technique directly predicts what content is visible within the 3D environment. By bypassing the conventional two-step forecasting process, this approach minimizes prediction errors and enhances accuracy.
The system conceptualizes the 3D space as divided into “cells,” treating each cell as a node within a graph network. It employs transformer-based graph neural networks to analyze spatial relationships among adjacent cells, while recurrent neural networks track the evolution of visibility patterns over time.
For pre-recorded VR experiences, this system can forecast visibility for users 2 to 5 seconds in advance, a major leap compared to prior technologies that could only predict a user’s FoV fractionally ahead of time.
“The time horizon in this work is particularly notable,” Liu commented. “Earlier systems managed to predict user visibility only for a fraction of a second, but our team has significantly extended that capability.”
The research team demonstrates a reduction in prediction errors of up to 50% for long-term forecasts, all while maintaining a real-time performance of over 30 frames per second—even when dealing with point cloud videos that include over 1 million points.
This advancement holds promise for consumers, offering a more responsive AR and VR experience with diminished data consumption. Simultaneously, it empowers developers to create sophisticated environments without the necessity for extraordinarily fast internet connections.
“We’re witnessing a shift where AR and VR are evolving beyond specialized fields into mainstream entertainment and everyday productivity applications,” Liu remarked. “Bandwidth limitations have been a significant hurdle, and this research provides solutions to overcome that issue.”
The research findings have been made available to support further exploration and development in this field. The study received backing from the US National Science Foundation grant 2312839.
Alongside Liu, the paper includes contributions from Chen Li and Tongyu Zong, both Ph.D. candidates at NYU Tandon in Electrical Engineering; Yueyu Hu, a Ph.D. candidate in Electrical and Electronics Engineering at NYU Tandon; and Yao Wang, a professor at NYU Tandon with dual appointments in Electrical and Computer Engineering and Biomedical Engineering, as well as roles at CATT and NYU WIRELESS.
Source
www.sciencedaily.com