Mixed Reality – revolution in medicine

Technological revolution in medicine, a doctor operating in mixed reality goggles has access to three-dimensional models of the patient’s organs – thanks to a project coordinated by Dr. Eng. Klaudia Proniewska from the Faculty of Medicine of the Jagiellonian University Medical College, a breakthrough in medicine is possible today.

KR: You use mixed reality in your project. What is this?
KP: Mixed reality is one of the so-called immersive realities, i.e. those in which we can delve into and perceive the world through a different perspective. Perhaps better known is virtual reality – completely computer-generated, or augmented reality that adds virtual elements to our world.
Mixed reality combines both of these worlds. It allows you to add virtual elements to the real world, but you can also interact with these elements. Putting on my mixed reality goggles, I can grab a computer-generated mug and place it on the table. It is this last type of reality that I focus on in my work.

What exactly is your project about?
Thanks to the Erasmus+ programme, we have purchased expensive HoloAnatomy® anatomy software that we can use in our laboratory. As part of other grants, the laboratory has been equipped with 10 modern technological workstations – each consists of a computer with Hololens 2 goggles. Now, thanks to the grant from the Erasmus+ project, we can focus on building a database of clinical cases for education.

The HoloAnatomy® software contains a complete library of holographic 3D human anatomy images, approximately 9,000 3D models, divided into female and male bodies. This allows us to create anatomy lessons, which makes it easier to view and learn about the human body in 3D.

In addition, thanks to advanced visualization techniques, we can compare the human anatomy in 3D with 3D models of a specific patient, which are prepared on the basis of imaging source data, for example computed tomography. Thanks to these models, students will be able to analyze individual clinical cases in greater detail and learn how to treat similar patients. We are also dealing here with personalized medicine. Importantly, our UJCM Mixed Reality Laboratory is the most modern laboratory of this type in Europe. We are the first in Europe to implement this software.

What is the process of creating such a data visualization?
Having a database of source images, i.e. computed tomography or magnetic resonance images, and having mixed reality at your disposal, such a database can be enriched with a three-dimensional model. For example, we have a patient with a liver tumor. Based on his tomography and MRI results, we can prepare an accurate model of this tumor, its location and a model of organs, because each person has a slightly different structure. After wearing the goggles, you will be able to see this 3D model, rotate it and see exactly what it looks like.

What was the inspiration for this idea?
It all started during my PhD studies at AGH. At that time, I went on an exchange under the Erasmus+ program to Europe’s largest cardiology center, Thoraxcenter, Erasmus MC, which is located in Rotterdam, where I learned about the secrets of imaging in interventional cardiology. It was in 2013 – already then there was more and more talk about three-dimensional methods in imaging, and it was within this subject that I conducted my research there. This way of presenting data makes it very easy to imagine the relationship between particular structures in space. In a two-dimensional image, it is difficult to imagine how exactly the artery runs, what diameter it has, and where exactly it may come from. Another direct inspiration was a video about modern mixed reality goggles and learning anatomy using HoloAnatomy® software.

Then I thought that I had to bring this modern equipment and software to Poland. And it worked.

Who, apart from doctors and medicine teachers, would like to use your knowledge and the laboratory?
This technology is used in many fields. We are currently conducting many informal talks and building a network of contacts. But we get a lot of inquiries about cooperation from clinics, schools, scientific circles, and the pharmacy department also approached us. These proposals reach us not only from Krakow.

Is mixed reality a revolution in teaching medicine?
I think so, and this is also indicated by the opinions of doctors, who are often also educators. We are currently in talks to introduce classes using mixed reality – in the form of optional classes – to study programs. Doctors want to show listeners what exactly clinical cases look like, and they say directly that they would like mixed reality to permanently enter the education curriculum.