When Apple announced the Vision Pro at its WWDC23 event on June 5th, it ushered in the era of spatial computing. Whether or not this takes off remains to be seen – Apple doesn’t plan to make the headset available on the market until sometime in early 2024 (for $3,499). That being said, the design is revolutionary and enables users to seamlessly transition from virtual (VR) to augmented reality (AR) with the goal of allowing users to remain connected to their physical and social environment. This event generated more excitement for the VR/AR community than any device or platform yet. It also led me to question where we stand in the clinical cardiovascular community when it comes to VR and AR technology? Will new and changing consumer technology, like Apple’s, again be a catalyst that drives healthcare technology forward (see SMART WARS: Atrial fibrillation vs consumer technology)?
Technology Primer:
- Virtual Reality: VR is a computer-generated simulation of a three-dimensional environment that can be interacted with in a seemingly real way. This interaction typically involves a user wearing a headset or a helmet with a built-in screen, sometimes combined with physical spaces or multi-projected environments to generate lifelike images, sounds, and other sensations. VR’s immersive nature allows users to experience different realities, from real-world simulations for training and education to fantastical environments in games and entertainment.
- Augmented Reality: AR is an interactive technology that overlays digital information—such as images, sound, text, or 3D models—onto the real world, enhancing the user’s perception of their surrounding environment. AR can be experienced through devices such as smartphones, tablets, smart glasses, and AR headsets. A common example is the popular mobile game, Pokémon Go, where digital creatures appear on the phone’s screen as if they were in the real world. Unlike VR, AR uses the existing environment and simply adds to it. By integrating digital information with our physical environment, AR provides a bridge between the digital and physical worlds, paving the way for exciting future developments.
- Spacial Computing: Spatial computing is an advanced form of human-computer interaction that integrates the physical world with the digital world, using the space around us as a medium to interact with digital information. Instead of being confined to flat screens, spatial computing leverages technologies like AR, VR and MR to overlay and embed virtual objects and data into our 3D world. It uses sensors, cameras, and AI algorithms to digitize the environment, allowing software to recognize and track objects and people in real-time. Users can then interact with the digital elements in an intuitive and natural way, whether through gestures, movements, or voice commands.
State of the Industry:
There are numerous potential applications for VR and AR in healthcare, and especially cardiovascular care. These include medical education, procedural planning / assistance, and telehealth augmentation. Despite this potential, active use of the technology is quite limited and mostly still investigational. Its primary application has been in the area of education and training: Stanford harnessed the Oculus Rift to guide students through anatomy lectures and lessons, the Heart Rhythm Society uses VR to help educate physicians in the steps of various procedures (see subcutaneous defibrillator implant), and there are numerous CPR training modules available in VR or AR format. Outside of education, the technology has primarily been used by the procedural subspecialties of Interventional Cardiology and Electrophysiology. VR and AR can help operators plan complex procedures by visualizing the anatomy and pathology of a patient in 3D and can incorporate data from other imaging modalities to personalize the experience to each individual patient. These systems can assist during the active procedure by overlaying relevant information on the field of view, such as echocardiographic images, CT images, as well as catheter and device position in real-time. VR is the modality commonly selected for education and planning purposes as it seeks to recreate the fully immersive environment or setting of procedural experiences. During the procedure itself, operators prefer AR as large portions require direct visualization of the patient, but certain aspects could be enhanced with additional data and imaging.
A number of studies have tested procedural AR on models and in the animal lab (see transcatheter aortic valve implant, transcatheter mitral valve repair) but few have used this technology during live human cases. In a rare example, researchers from the Electrophysiology Department at Washington University utilized Microsoft’s HoloLens AR technology during a live ablation procedure. During this case, local activation time mapping, catheter locations and lesion markers were all overlaid onto the patient’s cardiac geometry. The AR graphic was manipulated by two independent observers who were located outside of the lab but able to resize, rotate and adjust the geometry. That research lead to the creation of SentiAR, a medical technology company focused on developing clinical AR platforms for interventional cardiovascualr procedures. It’s initial product, CommandEP, displays an interactive 3D model of the heart created during the electroanatomic mapping portion of the procedure and displayed in real-time on the HoloLens.
There is slightly more experience with AR and VR in the surgical fields and it has been more widely assessed in that context. A systematic review of the topic in the neurosurgical literature is a few years old, but at the time almost all applications were similarly selective to training and education. A State-of-the-Art Review with a focus on Cardiology was recently released in JACC Cardiovascular Imaging, discussing many of these topics and reviewing the current literature.
Limitations and Growth Opporunities:
I have personally interacted with some of the VR technology available through the Heart Rhythm Society and for anatomic education at other institutions. There are a number of current limitations that contribute to the slow uptake of this technology, especially in the healthcare setting. Limited resolution, field of view, brightness problems, and prolonged latency are all holding current systems back, in my opinion, and are not sufficiently ready for active procedural assistance. Furthermore, the need for operator interaction with the patient, staff and local environment during a live case limit the use of fully virtual platforms while the state of augmented technology is not as well developed.
Many of these limitations may be about to change, thanks to the consumer technology and gaming industry. As already mentioned, Apple recently announced the Vision Pro and visionOS mixed reality headset. Among other features, the Vision Pro offers 180 degree field of vision, 23 million pixels (4K TV for each eye), ergonomic headset design and a revolutionary user interface that is controlled by a person’s eye glances, hands and voice (rather than clumsy hand controllers). Gaming platforms like Unreal Engine (Epic Games) and Niantic Labs (Pokemon Go) are rapidly advancing graphics resolution and AR capabilities for mixed reality. Unreal Engine has already partnered with PrecisionOS (an orthopedic simulation company) to develop ultra high-end virtual simulators for surgical training. We need much more innovative collaboration with companies like this to advance the field forward.
Summary (TL;DR):
- There is limited use of VR and AR in the cardiovascular care setting
- Current applications are crowded in the education, training and simulation spheres with rare examples of hands-on procedural application
- Training and educational tasks lend themselves well to fully virtual environments while procedural applications require augmented platforms
- Rapid advances in technology (Apple, Unreal Engine, Niantic Labs) for VR and AR are exciting and could open the door for accelerated partnerships and progress
Please subscribe to our Newsletter for email notifications containing new posts as soon as they are published:
Cardio Cypher 