A doctor wearing virtual reality (VR) goggles,
A doctor wearing VR (Virtual Reality) goggles performs surgery while referring to a 3D organ model of the patient floating in the space in front of him or her and the surgical plan.
By introducing cutting-edge technology to the frontlines of medical practice,
The company continues to take on the challenge of solving issues such as the shortage of doctors and regional disparities in medical care by introducing cutting-edge technology to the frontlines of the medical field,
Okinaga Research Institute of Teikyo University Innovation Lab's Maki Sugimoto Professor.
While working as an active Surgery doctor in the clinical field,
While working in the clinical field as an active physician, he launched his own start-up company to implement the results of his research into society as quickly as possible.
How is Sugimoto Professor, who has the perspectives of a physician, entrepreneur, R&D developer, and educator, trying to pioneer the future of medicine?
Sense of crisis over the current state of exhausted doctors
Optimizing medical care with digital technology
Surgery wearing goggles are performing surgery in an operating room. To those without the goggles, it looks like a normal surgery, but in front of Surgery' eyes, 3D models of the patient's lesions and organs are floating. The surgeons can grasp and rotate the floating 3D models of tumors, blood vessels, and other organs to check the blood vessels behind the organs as they proceed with the surgery.
It's like something out of a science fiction movie, but this kind of scene is already a reality. Professor Sugimoto, Surgery performing the operation, is also the founder and CEO of a startup company developing this medical VR software. Professor Sugimoto's motivation for seriously considering the introduction of VR came from his time at Teikyo University Chiba Medical Center, where he had been working since 2004. Doctors at regional hospitals were extremely busy, and seeing their exhausted workforce made him feel a sense of crisis.
At the time, endoscopic surgery was starting to spread nationwide, and it was attracting attention as a minimally invasive surgery that reduces the physical burden on patients. This type of cutting-edge medical technology was preferentially introduced in large hospitals in urban areas, but doctors in rural areas had no opportunity to be exposed to the latest technological updates. Moreover, they were so busy dealing with the daily influx of patients that they didn't have the time to go to a hospital with the equipment to learn the techniques.
"The biggest problem was that doctors' motivation was dropping because they were too stressed. By introducing cutting-edge technology, we wanted to make their work more efficient and help them find motivation in their work again," Professor Sugimoto recalls.
Developing and commercializing self-programmed medical devices
Making VR technology easily available from the cloud
So they started working on creating 3D data from diagnostic images. By creating 3D data of each patient's lesions and organs, it becomes possible to grasp the "depth" of tumors and blood vessels during surgery. In Professor Sugimoto's specialties of hepatobiliary, pancreatic, and gastrointestinal Surgery, depth information during surgery is particularly important, as blood vessels and other structures run intricately within organs and surrounding tissues. The intertwined three-dimensional structure of these structures varies greatly from person to person, and the extent of cancer in particular is completely different for each patient.
"If we could operate while referring to the patient's own 3D image like a car navigation system, we could see the back of the organs and the areas that need to be removed, avoid damage to blood vessels and other organs in advance, and accurately reproduce the surgical plan. At the time, I discovered that commercially available 3D medical image software could also display CT images as 3D digital data on my laptop, so I started using it during surgeries. When other Surgery heard about this, one after another they said, 'We want to try it too,' and it spread throughout the hospital. As surgery became more efficient, the motivation of my fellow Surgery increased, which made my job more rewarding." (Professor Sugimoto)
However, he gradually realized that simply displaying 3D data on a computer screen was not enough to understand the three-dimensional relationships between organs and lesions, and wanted to understand the true anatomy in three dimensions.After researching VR technology, he decided to use inexpensive, readily available, commercially available hardware rather than expensive specialized equipment, and focus on software development to help many Surgery is how "Holoeyes MD" was born.
Holoeyes MD
The service is in the form of SaaS (Software as a Service), which allows users to use the app by accessing it via a network. Once data created from 3D images such as CT or MRI is uploaded to the cloud, a three-dimensional model of the organ can be viewed in VR space five minutes later. The VR goggles are equipped with sensors that capture hand movements, and can detect hand and finger movements to rotate the generated model, and can also display cross-sectional images from the menu screen. While it is not possible to operate a mouse in the clean surgical field during surgery, the VR goggles make it possible to freely operate the 3D model even with sterile gloved hands.
From nurse to engineer
Helping medical professionals and patients through innovative technology development
Professor Sugimoto founded Holoeyes Inc. in 2016. "Holoeyes MD" was certified as a medical device in 2020 and is now used in many medical institutions. The service also includes an optional feature called "Holoeyes VS," which allows multiple users (Surgery) connected via a network to view the same 3D model data and share and conferencing surgical techniques. This allows users to learn actual surgical techniques in a realistic and efficient manner by superimposing the movements and gaze of veterans, which have previously been considered tacit knowledge, onto their own bodies.
In addition, the company offers an educational app called "Holoeyes Edu," which records the hand movements and audio commentary of veteran doctors during surgery on the cloud and allows users to re-experience the surgery through a smartphone and simple VR goggles sold at a 100-yen shop. This app is also being used in medical education for medical students, nursing students, and others.
The Innovation Lab Okinaga Research Institute has been conducting joint research with many companies, including Holoeyes, and has been collaborating with industry, government, and academia. Assistant Professor Takumi Sueyoshi of the Innovation Lab is currently working on the research and development of new technologies. Assistant Professor Sueyoshi was originally a nurse assigned to the operating room, and met Professor Sugimoto when he was a new nurse.
"When I began to train junior nurses as a mid-career nurse, I realized the limitations of traditional manuals. There were video teaching materials, but they were difficult to understand for new nurses. That's when I remembered seeing Dr. Sugimoto's VR surgical support in the operating room as a new nurse, so I decided to attend a vocational school while working as a nurse to learn VR software development." (Assistant Professor Sueyoshi)
She then decided to quit her job as a nurse and began working with Professor Sugimoto at Teikyo University to develop innovative technology. Professor Sugimoto also gives her stamp of approval, saying, "She is an indispensable asset in the development of this technology, presenting our research results at academic conferences overseas."
"As a nurse, I interacted with each patient individually, but now, as an engineer, I'm in a position to help medical professionals. By doing so, I'm able to help doctors, and ultimately help more patients," says Assistant Professor Sueyoshi.
Expanding possibilities in medical care and education
Spatial Computing
While futuristic operating rooms are becoming a reality thanks to VR goggles and 3D holograms, Professor Sugimoto has his sights set on a future even further: first, making this technology, which is currently being used in around 60 medical facilities, available to more medical facilities.
"You can really appreciate the benefits of this technology once you experience it. However, Surgery tend to have a craftsman-like mentality, and it's traditional for junior surgeons to hone their skills through the same hard work as their seniors. I want to change that perception. Through technology like this, we should be able to create an environment where veterans and junior surgeons can learn from each other." (Professor Sugimoto)
"Spatial Computing" is a technology that integrates physical space and digital information, including VR and AR (Augmented Reality), to enable interactive experiences. Utilizing this technology, which seamlessly connects the real world and digital content, the possibilities of medicine and medical education are expanding, and recently research has been promoted into diagnostic systems that incorporate machine learning and AI into VR technology.
However, Professor Sugimoto emphasizes that we must not forget that both patients and doctors, who are the parties involved in medical care, are human beings. "By continuing to work in the medical field, we can understand the issues that need to be resolved, and we want to implement innovative technologies to solve these issues as quickly as possible in society and continue to create new social value. We have nicknamed this lab "ILORI," which is an acronym for the English name "Innovation Lab, Okinaga Research Institute." We want this lab to be an open base where people gather and ideas are born, just like the ancient Japanese "irori" hearth." (Professor Sugimoto)
University research institutes have the mission of broadening the horizons of science through basic research, but this lab, which bears the word "innovation," places emphasis on social implementation. Professor Sugimoto believes that speed is also important for this purpose. They are always running at top speed, slightly ahead of the times.
