Ground control to major surgery

Technologies that once remotely guided the Mars Rover millions of miles from Earth are being used to create a virtual reality platform to remotely train medical students and revolutionise how surgery is observed.

It is part of a project between a machine vision company and the University of Sheffield Advanced Manufacturing Research Centre (AMRC).

Dr Ben Crutchley, senior machine vision engineer at i3D robotics, (i3Dr) explained: “The Covid-19 pandemic has highlighted the global need for a digitally-connected world that allows for seamless remote working conditions.

“The ability for senior surgeons and doctors to remotely view and be immersed in a virtual representation of an operation will have a game-changing impact.”

i3Dr worked with engineers at the AMRC, part of the High Value Manufacturing (HVM) Catapult, on a bid for funding from Innovate UK for the Stereo Theatre project.

The money, part of the Fast Start competition, is a share of a £40m Government investment to advance innovative projects which tackle economic and societal issues resulting from the Coronavirus crisis.

Stereo Theatre will build on the work already done by the Medical AMRC in Rotherham, where an Industry 4.0 Digital Operating Theatre proof of concept has been built.

The demonstrator combines a virtual reality digital twin, projection mapping and smart tools that enables the position of objects and clinicians to be accurately tracked in the theatre space, with relevant information displayed digitally using screens, projections and augmented reality (AR) devices.

Crutchley said: “Currently, the AMRC’s Digital Operating Theatre can provide a virtual representation of the real-world theatre and monitor movements with commercial off-the-shelf sensors and smart tools, but does not have the ability to produce updated 3D models of patients.

“Stereo Theatre will fill this gap, allow for further upscaling of the technology and offers a revolutionary approach to both the teaching of medical students and surgical procedures.”

Both i3Dr and the AMRC believe the technology could transform surgery and teaching by enhancing the way medical students can learn remotely, performing simulated surgeries on a virtual reality training platform.

The Stereo Theatre project also enables senior consultants to be engaged in surgical procedures conducted by junior colleagues remotely, meaning more patients being observed by experienced surgeons.

Digital design manager in the AMRC’s Design and Prototyping Group, David King, said: “When the AMRC started the Digital Operating Theatre project our vision was to develop a fully immersive real-time digital twin of the surgical environment that would be as realistic as possible.

“The integration of i3D robotics real-time stereo mapping technology within the AMRC Digital Operating Theatre brings us closer to our vision and provides a solution to the need for accurate real-time 3D visualisation of patients within the digital twin environment.”

i3Dr, based in Tonbridge, specialises in producing hardware and software solutions for industrial environments such as steel, nuclear and aerospace.

Its core technology, stereo vision, has evolved through decades of development for the Mars Rover missions.

The Stereo Theatre simulator will be tested in the Medical AMRC’s Digital Operating Theatre demonstrator to prove the technology can monitor the real-world patient to update a digital twin displayed in VR and AR in real or near-real-time.