Design Validation Part II: Beta Testing – Fine-tuning Design To Create The Ideal Safety Scalpel
January 11, 2020Throughout the design and development of the Anzen Safety Scalpel, the feedback from the end-users – surgeons and surgical technologists, was vital to ensure that the end product would be something surgical professionals would use in the operating room.
The Anzen Safety Scalpel went through 2 rounds of user testing – alpha testing and beta testing, before the design team was confident that the product was ready for market. During alpha testing, the feedback from surgeons was that the blade was not sharp enough. Surgeons also had trouble adjusting to a new device in the operating room.
As such, the design team conducted blind blade studies from different blade suppliers and selected new blades based on the feedback from the blind blade studies. In addition, to increase surgeons’ comfort level using the Anzen Safety Scalpel, the evaluation process was modified to ensure all users familiarize themselves with the device via simulated use testing first, before actual use testing. This would ensure that surgeons are not blindsided by the use of a new device that they had no prior experience with while in the operating room.
Blind Blade Study – The Search For a Sharper Blade
Before moving onto beta testing, the Anzen Safety Scalpel design team needed to make sure that the surgeon end-users would be happy with a new blade supplier. But which one would be sharp enough for our surgeons? To find out, the design team conducted blind blade studies and selected the manufacturer(s) that surgeons rated as the sharpest out of 5 different sources. Click here to learn more about the blind blade study process and how the new supplier was eventually selected.
Beta Testing
After the new blade supplier(s) were selected, the second part of the design validation was ready to begin. Beta testing was conducted at 28 different facilities between 2018 and 2019. Similar to the alpha test, both simulated use and actual use testing were carried out.
Surgeons were asked questions that related to the overall ease of use, blade sharpness, use of the device to make incisions, and the feel of the device compared to traditional scalpels. Surgical Techs were asked questions that related to the overall ease of use, attaching/detaching the cartridges onto/from the handle and passing of the device to the surgeon. The questions asked were similar to those asked during alpha testing, to facilitate comparisons between the 2 phases. Questions were scored between 1 to 5 where 4.0 and above considered positive, 3 was a neutral response, and any score below 3 was considered a negative response.
During simulated use testing, surgeons and surgical technologists were asked to familiarize themselves with the use of the Anzen Safety Scalpel by making incisions on a simulated skin pad. Users then filled in a survey form. There were 17 questions asked during simulated use testing, and all questions were rated positively by surgeons and surgical technologists with a rating of 4.0 or higher.
After the surgeons and surgical technologists were familiar with how to use the device, they then proceeded to actual use testing where the Anzen Safety Scalpel was used in patient procedures. Surgeons and surgical technologists were asked to fill in another survey after their procedures were completed. Out of the 11 questions on the actual use survey form, all 11 questions were rated positively with a score of 4.0 or higher.
A Comparison Between Alpha Test and Beta Test Results
The main concern surgeons had during alpha testing was the sharpness of the blade. How did the sharper blades from the new supplier that were used in beta testing compare with those used in alpha testing?
2 questions relating to blade sharpness were asked during both alpha and beta testing. Results show that surgeons were pleased with the sharper blades and rated them positively.
In addition to the blade sharpness, the design team also looked whether there were any changes in the overall impression of the Anzen Safety Scalpel from alpha test to beta test. Results showed that the overall impression of the Anzen Safety Scalpel remained the same from alpha test to beta test for simulated use. However, during actual use, the overall impression of the Anzen Safety Scalpel improved in beta testing compared to alpha testing.
This could be due to the change in process, requiring surgeons and surgical techs to do simulated use testing before actual use testing to ensure they were not blind sided by a new device in the OR, in addition to the sharper blades.
The Anzen Safety Scalpel – What Next?
Breaking down the user testing into 2 parts, alpha and beta testing, allowed the design team to incorporate feedback from alpha testing into the Anzen Safety Scalpel to make improvements before gathering additional feedback on the improved design.
The results from the beta testing show that the requirements for both simulated and actual use testing were met during beta testing. In particular, the concerns with blade sharpness that were brought up in alpha testing have been resolved with the sharper blades from the new supplier. Overall, surgeons and surgical techs are happy with the final device and the Anzen Safety Scalpel is now ready to launch.