Thesis Description & Objectives
The integration of Large Language Models (LLMs) with Virtual Reality (VR) offers promising opportunities to redefine how users interact with immersive environments. This thesis explores the use of LLMs to support more natural, intuitive, and hands-free interactions in VR, moving beyond traditional input methods like physical controllers.
The primary goal is to design and implement a framework that leverages the language understanding capabilities of LLMs to enable voice-driven interactions within VR environments. Potential use cases include hands-free locomotion (e.g., teleportation or virtual walking via spoken commands), interaction with virtual objects or interfaces through natural language, and the integration of multimodal LLMs to interpret visual inputs, such as referencing objects in the environment through verbal cues.
The work will involve defining the system architecture, developing prototypes using Unity and voice processing tools, and exploring how LLMs can interpret and respond to user input in real time. The system will be evaluated through user studies, collecting both quantitative and qualitative data to assess usability, command recognition accuracy, response times, and overall user experience.
References:
[1] D. Calandra, F. G. Pratticò, and F. Lamberti, “Comparison of Hands-Free Speech-Based Navigation Techniques for Virtual Reality Training,” in 2022 IEEE 21st Mediterranean Electrotechnical Conference (MELECON), Palermo, Italy, 2022, pp. 85-90. doi: 10.1109/MELECON53508.2022.9842994.
[2] J. A. V. Fernandez, J. J. Lee, S. A. S. Vacca, A. Magana, B. Benes, and V. Popescu, “Hands-Free VR,” arXiv preprint arXiv:2402.15083, 2024
[3] S. Özdel, K. B. Buldu, E. Kasneci, E. Bozkir “Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Reality” arXiv preprint arXiv:2504.17331, 2025
