UX/ UI and Product Designer
Navar
¿What it's all about?
This is a case study of a visual guide for RLCA using augmented reality (AR). This guide provides guidance to the orthopedic surgeon by employing anatomical, strategic, and operative information necessary for the correct location of the insertion and exit points of the tibial and femoral tunnels, using augmented reality, which is an emerging technology that provides graphical information superimposed on a real environment by means of a head-mounted device (HMD).
Drill angulation system
My role
My role as an industrial design student along with that of my partner consisted of:
-
Literature analysis regarding the RLCA process, and augmented reality tools among others.
-
User analysis, surveys, card sorting, etc.
-
Design of the user interfaces for the augmented reality glasses.
-
Design and execution of usability testing for the selection of interaction techniques and testing of visual guidance with orthopedic surgeons
-
Communicating design needs with systems engineers
Participants
-
Andrea Paola Pinzón Barbosa
-
Sara Maritza Gutiérrez Rondón
-
Luis Eduardo Bautista Rojas
Director
Institutions
Problem definition
¿How does proper visualization help to improve the accuracy of placement of insertion points for tibial and femoral tunnel drilling?
What is ACL reconstruction?
Anterior cruciate ligament reconstruction (ACLR) is a common operation in orthopedic surgery to restore stability to an injured knee at the anterior cruciate ligament (ACL).
Once the anterior cruciate ligament is partially or completely torn, it does not self-regenerate like a broken bone. Therefore, a ligament replacement is necessary to regain knee mobility. For its replacement, the patient's own tendons (autografts) or tendons from a bank (allografts) can be used.
¿How does it occur?
Anterior cruciate ligament injury occurs when the femur and tibia perform a twisting motion in opposite directions.
Some facts
-
70 to 80% of unsatisfactory cases in RLCA are due to technical errors in the performance of the tibial and femoral tunnels.
-
In more than 10% of cases of ACL reconstruction, unsatisfactory clinical results are presented.
-
The most frequent failures in the realization of tunnels for the RLCA are due to the fact that there is very little anatomical information that allows the correct location of the natural footprints of the LCA.
OK: let's get down to business
Design proposal development
But first, Ideation
According to the task, information, and interaction diagram, we started with the creation of different sketches that would show the information needed during each process of the workflow.
These sketches were corroborated with the systems engineers to identify the feasibility of the drill angle system.
Hands-on:
First virtual prototype in Meta® glasses
Initially, a virtual prototype is made, in order to have knowledge of the workspace that allows Meta® glasses (800x350 px), verify the size for the readability of the elements and the functionality of the indicators, then a brief description of the first virtual prototype will be made.
The information displayed on the first prototype of the Meta® augmented reality glasses:
-
Bones, tibia and femur in transparent.
-
Entry point, exit point, and safety zone in a spherical shape.
-
System for angling the tool similar to that mentioned visualization systems Percutaneous needle intervention (Peterhans, Kim, & Weber, 2012).
-
Visualization of the tool.
-
Guide tunnel.
-
Buttons for direct change from the femur to the tibia.
-
Bar indicating that the tool is correctly positioned for drilling.
-
Indicators in the change of appearance: both the color of the tunnel, the entry and exit point, and the bar change color according to the position of the tool.
-
Drilling progress.
According to the tests performed with the first prototype, it was observed that the visualized information allows the identification of the proposed elements. However, in the case of the virtual models of the patient's knee, the use of transparency means that the depth and dimensions of the bone are not correctly perceived. In the case of the functionality of the indicators, the change of appearance allows to know that the action has been performed correctly or incorrectly, likewise, the use of the target allows one to recognize the necessary movement to angle the tool with respect to the exit point when the guide pin is in the pivot with the entry point.
Second virtual prototype in Meta® glasses
In the first prototype, the readability of the objects was evaluated through inspection, and for the design of the second prototype the use of color, the distribution of the elements in the workspace, and the use of diagramming for the content of the interface were taken into account, without forgetting the considerations of the first prototype.
A layout was made with the law of thirds and the indicators were distributed on the screen in order to optimize the workspace, however, at the time of locating points, the user's focus is reduced due to concentration, which generates the other indicators are not visualized despite being in the field of vision. Finally, for the query information found in the side menu, it was identified that it can become annoying and cause distractions, therefore, it is proposed as a pop-up menu.
In between: some other alternatives
To get: Final virtual prototype in Meta® glasses
Based on the considerations of the second prototype, the following modifications were made: The indicators were transformed to create a system and not a set of them. This system will be attached to the drill bit and not fixed to the glasses, the only information attached to the glasses will be the tunnel and the pop-up menu manipulated by the Myo® bracelet.
Three possible target states are shown, which represent the location of the bit tip with respect to the entry point, the vertical line represents the pivot movement from left to right, and the horizontal line represents the up and down movement. When the intersection point of the lines coincides with the center of the circumference, it means that the angle of the drill bit coincides with the two points, the entrance and exit points of the tunnel, therefore, it can be drilled. This is also reflected in the change of color of the triangles that are around the circumference, when they are red, it indicates that the drill bit is NOT correctly positioned, and when it is green, it indicates that the drill bit is in position to start drilling.
To know the depth and travel of the drill bit, 3 indicators are observed, a numerical one, which tells me the drilling depth, and the pentagon next to the color change of the circumference perimeter, both when in a "full" state, indicates that the drilling has been completed.
How did we reach this result?
Good news, it's not boring for me!
So, for the development of this project, it was necessary to perform a literature search to find information that provides fundamental data on knee anatomy, augmented reality, and human-computer interaction in the framework of medicine. For this purpose, a 4-step search equation was designed with the next steps:
-
Unstructured exploration
-
Grouping of similar terms
-
Combination of terms
-
Search equation
After exhaustive research we found that
-
There are three fundamental components of image-guided surgery: data, visualizations, and views.
-
Fundamental factors of positioning and angulations of the guides for correct drilling.
-
There are several augmented reality systems used in surgical as well as training fields.
-
Identification of common elements of the existing systems.
The literature analysis allowed us to recognize and select information considered relevant to the workflow in the RLCA surgical procedure, this classification was carried out with the users through interviews and the use of card-sorting.
let's organize the data!
Information Architecture
But first :
¿What data is required for the design of a visual guide in the ACLR?
According to the information gathered and analyzed in the literature review and the inspection of the mentioned state of the art, a series of information was selected, which is considered relevant to the workflow in the surgical procedure of the RLCA, then classified into anatomical, strategic, operative, general, patient and raw image data (raw image data are the images obtained from the patient that are not processed in any way, such as CT scans and X-rays).
Which of this information is the most important?
Hierarchization of information
In order to solve this mystery, we turned to our users.
A survey of orthopedic surgeons was carried out, in which a list of items identified in the previous section, which refer to types of information that may be required by the surgeon in the operative procedure, was disclosed, and the user was asked to rank the items according to the following levels:
- Primary or essential information,
- Secondary or useful
- Tertiary or supportive
With the intention of finding out which information has the highest degree of importance.
And, ¿what about the users?
User requirements
The requirements were established by means of an interview with the direct users, where the general aspects of the surgical intervention and the user's needs in the procedure were inquired.
such as:
-
Surgical elements
-
Surgical intervention process
-
Minimum information required during the surgical procedure
Now: ¿Why a head-mounted device HMD?
Selection of the display system
For this stage, it was identified that for real-time visualization of the data, there are different technologies and electronic devices that allow the surgeon to perform the surgery (Sielhorst et al., 2008).
HMD
Projections on the skin
Assisted surgery
The following table shows seven means by which structured information is projected, in which hand-eye coordination, image fusion, implicit 3D interaction, stereoscopic visualization and multi-user capability are evaluated. The vast majority of navigation systems for the operating room make use of monitors or tablets since they are less expensive and do not present as many limitations as skin projections or the implementation of virtual guides directly on the arthroscopic image. However, these visualization systems are not efficient (Preim & Botha, 2014).
Therefore, the HMD visualization system has been proposed, different from augmented optics, since the information is displayed on the two lenses as if there were a screen in front of the user's eyes (Abe et al., 2013). This system allows an improvement in hand-eye coordination by having in the same place the information that guides the surgeon to perform the tasks required in the flow of the operation, it also allows image fusion, contains implicit 3D manipulation; provides stereoscopic visualization, and gives the option of recording from the user's perspective. As the device is located on the head, the images are projected at the user's viewing angle and are not affected by head movement; the size of the device and its location frees up the working area and gives accessibility and mobility to both the surgeon and his team. Therefore, these properties make the HMD one of the most complete and adaptable configurations for the surgical environment.
¿How users are going to manipulate it?
Selection of the interaction technique
An important factor is the autonomy of the surgeon during the surgical procedure. Therefore, when employing a visualization on an HMD device it is necessary to select an interaction technique that allows manipulation of the graphical user interface to be projected (Wiberg, 2016).
Thus, it is required to identify the interaction technique that suits your needs; that allows compliance with safety and hygiene standards in the operating room since the surgeon must perform hand washing before entering the operating room, to reduce bacterial flora.
Malignant or pathogenic microbes and bacteria, by contagion (direct or indirect), can become installed in the patient's organism in sufficient numbers to produce an infection (Carlos, 2012), this prevents him from having contact with objects foreign to the surgical procedure, such as handling screens, keyboards, mouse, etc.
Device evaluation
An evaluation of the Myo® device and gestural interaction boards in the Meta® glasses space was conducted to determine the manipulation of the graphical interface in the ACLR according to the following usability criteria:
- Effectiveness
- Efficiency
- Satisfaction
The experiment consisted in performing 3 activities manipulating the different interaction techniques. In both cases, the relationship between gestures and actions was revealed.
Test with the MYO bracelet
Test boards in space
The Myo® device was selected for the manipulation of the graphic interface visualized in the HMD augmented reality system since it is the tool that showed the lowest degree of difficulty in memorizing the gestures, the shortest time in the development of the tasks, and the least amount of errors by the user.
¿Which route to take?
Workflow of the surgical process
To design the interface, it was necessary to identify the workflow executed by the surgeon when performing the specific task. For this purpose, closed card sorting was carried out with orthopedic physicians, to know the workflow and thus organize the information (Yusef et al., 2006).
During the development of this activity, the surgeon was given nineteen cards with names of the actions required for tibial and femoral tunnel drilling. The user's job consisted of arranging these cards consecutively according to the sequence in which they usually perform them.
¿And, does it work?
Testing
The evaluation of the graphical interface for the Meta® augmented reality glasses, together with the implementation of the LED screen orientation system, was carried out in order to determine whether the attributes of the two graphical interfaces affect the quality of the user experience during tunnel drilling for anterior cruciate ligament reconstruction RLCA, according to the following usability criteria: effectiveness, efficiency, and satisfaction, through the design and execution of a test.
Performance of the test
Description of the activities
The activity consists of the location and drilling of the tibial and femoral tunnel simulating the reconstruction of the anterior cruciate ligament.
for this process, artificial bones were used.
Development of the activity
The evaluation process began with training the orthopedic surgeons on the indicators found in the interface, using the projection of a video that demonstrates the indicator-information relationship.
Then we proceeded to explain to the participant the task to be performed, the creation of tunnels for the reconstruction of the anterior cruciate ligament, this activity was divided into:
-
identification of the instrument to be drilled,
-
Identifying and locating the entry point of the tunnel, and
-
the orientation of the drill bit according to the shown system
-
finally, the identification of the drilling progress.
Usability metrics
Three forms were used to collect information, one for efficiency, effectiveness, and satisfaction.
The first one consists of the time taken for each task if the participant manages to perform it at the first attempt, and if not, the number of attempts made. The second consists of the number of errors, interventions, and requests for help and finally, for satisfaction, a survey was carried out in which the participant rated the elements of the graphic interface from 1 to 5, 1 being the lowest level together with the EVA rating of the system as a whole.
¿would you like to know more about the project?
If you want to know more about this project and the evaluation method, you can download the article (available in Spanish) 👇
Conclusions
-
The methodological development of the project was accompanied by orthopedic surgeons in each of its phases, which allowed for adequate check the decision-making throughout the design, development, and implementation of the visual guide.
-
The design of the guide was developed continuously, it was a process of constant improvements, under the premise of providing the orthopedic surgeon with pertinent information in an accessible way at the moment of performing the tests for the surgical process of tibial and femoral tunnel perforation in the RLCA.
-
The importance of iterative GUI prototyping was evidenced in the development process, as this allows for continuous evaluation and optimization of the GUI proposals.
Awards
