These past two weeks have taken me out of my comfort zone and into the world of AR/VR—a field I had little personal experience with until now. I was surprised to discover how much these technologies have advanced in recent years, driven by competition between Meta and Apple as they compete to define the future platform for computing.
Before developing any VR project, I first explored its history. The form factor of a head-mounted display has seen a resurgence, but only time will tell if it will remain or fade again into obscurity. The history of VR can be traced down to Cybernetics, introduced by Wiener (1950), focusing on feedback mechanisms and adaptive systems. Unlike AI, cybernetics emphasizes human-machine interaction, forming continuous feedback loops that create immersive experiences.
During our lecture on VR concepts, two core ideas stood out: immersion and embodiment. Immersion is achieved through head tracking and spatial audio, replicating natural stimuli to enclose the user in a virtual environment. Embodiment enables users to interact with the virtual world as if it were their own physical body. These elements create a convincing illusion of reality, raising philosophical questions about the nature of perception and reality.
Tracing the historical connections in Silicon Valley, I examined Ivan Sutherland’s Sword of Damocles and its influence on modern VR/AR headsets. Additionally, Douglas Engelbart’s 'Mother of All Demos' introduced foundational computing concepts like hypertext and video conferencing, emphasizing augmentation to enhance human capabilities. These historical perspectives shape my approach to designing for immersive experiences.
My first experience with the Meta Quest 3 was thrilling. Using hand gestures to manipulate virtual screens felt futuristic. However, reflecting on Engelbart’s vision, I questioned whether this interaction truly augments productivity and collaboration. The lack of eye tracking in the Quest 3 stood out to me, as eye tracking provides an accessible and intuitive input method. Drawing from my experience with TensorFlow.js for gaze tracking, I realized that infrared-based tracking would greatly improve accuracy.
For my project, I explored using the Meta Quest 3’s head-tracking system to highlight words as users read. To implement this in Unity, I positioned an invisible sphere in front of the user’s face and detected intersections between the sphere and text objects. This approach mimicked eye tracking by dynamically adjusting text appearance based on user gaze.
Initially, I struggled with manipulating text in Unity’s canvas. Through collaboration with a more experienced student, I found a workaround: instead of modifying text within the canvas, I created individual TextMeshPro objects for each word. This allowed greater flexibility in dynamically changing text properties based on user interaction.
In retrospect, my project lacked nuanced interaction design. Moving forward, I plan to dedicate more time to sketching interactions before development. Eye tracking, rather than head tracking, would provide a more natural reading experience. Exploring external solutions, such as the EyeWriter project, or using a more advanced headset could improve accuracy and usability.