Thesis Description & Objectives
This thesis explores the design and development of a prototypal framework for integrating multimodal perception, reasoning models, and agent orchestration in XR applications. These capabilities should be exploited to build spatially aware, multimodal AI agents that deliver low-latency, context-aware assistance in XR experiences. The system should make use of the most recent available AI models and frameworks, such as NVIDIA XR AI, exploring three core concepts:
• Ingest real-world signals from XR devices, including video, audio, depth, pose and sensor data.
• Connect agents to specialized tools and services for video search and summarization (VSS), for visual AI and video understanding, and for knowledge retrieval and retrieval-augmented generation.
• Support multiple AI models.
