Experts Weigh In: Common Pitfalls to Avoid in Usability Testing
UserWise, a ClariMed Company, recently participated in an expert panel discussion hosted by the San Francisco Bay Area Biomedical Engineering Society focusing on important questions companies encounter when trying to apply Human Factors and Usability Engineering to Medical Devices. A variety of questions from the audience and useful tips from the experts’ own experience were discussed in detail. This is the second installment in a series of posts on Usability Testing and the Human Factors product submission process summarizing discussions from the panel. For more details on the panel discussion regarding when in product development to usability should be considered for a medical device, read the first installment in the series. In this post we'll answer the following question:
Numerous considerations for conducting usability validation testing were named by both the panel and the audience. The most important pitfalls to avoid are highlighted below:
Lack of moderator training or lack of experience managing usability testing can also be detrimental to the Human Factors submission. Many companies try to combine their usability study with their clinical study to save money. If this strategy is utilized, care must be taken to ensure the people moderating the study are appropriately trained in usability best practices. The moderator of a usability study must not only be trained to avoid introducing bias but he or she must also be trained and prepared to react to unexpected behaviors from the participant during testing. The moderator must be able to recognize when a use error has occurred and how to probe the participant appropriately to determine the root cause of the use error. The FDA and notified bodies expect root causes to be delineated in the study report. UserWise stressed the importance of correctly probing the participant without introducing any form of bias. If bias is introduced to the participant while probing to find the source of the use error, then the resulting information may not be accurate or valid. This can have a negative effect on the Human Factors submission and require additional clarification or data to support why the information is still representative, or worse, it may result in the need for an additional supplemental study. Having a trained moderator with experience running usability testing and knowledgeable in not introducing bias is the key to the success of your usability study and submission.
Moderator behaviors are not the only source for bias, study environments also have the potential to influence the participant. Usability validation testing must be conducted in an actual use environment or in a simulated use environment. When simulated use environments are used, they must be representative of the actual use environment because the environment can strongly influence the user through factors such as lighting, noise level, workspace size, workspace clutter, and sanitation hazards. Environmental bias can be much harder to recognize, so properly simulating a test environment is critical to getting accurate results from usability testing. This means surrounding the study participant with the fixtures, equipment, lighting, and sounds typically found in the places where the device is used. If the environment is improperly prepared, then the study results may not fully represent intended use.
Another possible source of bias for a usability study is over-training the participant. The training provided to the participant during the usability testing must be representative of what will be done in the field. Over-training the participant is considered a form of study bias. If users are typically provided an overview of the product by the manufacturer, then the usability study should simulate this. If training is not typically provided to users in the field, then training should not be provided during the usability testing. Training given to the study participant should also be given in a manner expected of the participant’s job title or responsibilities. Training for a cath lab technician using a product may be different than training provided to a clinician using the same product. If users are not expected to use the medical device immediately after training, then a training decay must also be implemented during the study. Typically, participant must have a minimum training decay period of one hour; however, more may be warranted depending on the situation. The panel stressed that this is one area where people inexperienced in human factors engineering and usability testing may unknowingly introduce inappropriate bias into a study. Not allocating a training decay of at least one hour prior to the study can invalidate the usability testing for the Human Factors submission. So, be sure to provide representative training to participants during your usability study.
If you, or your organization would like further information on how to properly conduct usability testing, we are happy to help. UserWise is experienced at conducting usability studies and successful Human Factors Submissions and is prepared to assist you with any questions. Reach out to us on our contact page.
︎Garrett Murray & Denise Forkey | December 20, 2017
What are important pitfalls to avoid when conducting usability testing for a medical device?
Numerous considerations for conducting usability validation testing were named by both the panel and the audience. The most important pitfalls to avoid are highlighted below:
- Introducing cognitive bias
-
Failing to probe for root causes
-
Simulating the intended use environment incorrectly
- Providing training to participants that is not representative of training provided in the field
Lack of moderator training or lack of experience managing usability testing can also be detrimental to the Human Factors submission. Many companies try to combine their usability study with their clinical study to save money. If this strategy is utilized, care must be taken to ensure the people moderating the study are appropriately trained in usability best practices. The moderator of a usability study must not only be trained to avoid introducing bias but he or she must also be trained and prepared to react to unexpected behaviors from the participant during testing. The moderator must be able to recognize when a use error has occurred and how to probe the participant appropriately to determine the root cause of the use error. The FDA and notified bodies expect root causes to be delineated in the study report. UserWise stressed the importance of correctly probing the participant without introducing any form of bias. If bias is introduced to the participant while probing to find the source of the use error, then the resulting information may not be accurate or valid. This can have a negative effect on the Human Factors submission and require additional clarification or data to support why the information is still representative, or worse, it may result in the need for an additional supplemental study. Having a trained moderator with experience running usability testing and knowledgeable in not introducing bias is the key to the success of your usability study and submission.
Moderator behaviors are not the only source for bias, study environments also have the potential to influence the participant. Usability validation testing must be conducted in an actual use environment or in a simulated use environment. When simulated use environments are used, they must be representative of the actual use environment because the environment can strongly influence the user through factors such as lighting, noise level, workspace size, workspace clutter, and sanitation hazards. Environmental bias can be much harder to recognize, so properly simulating a test environment is critical to getting accurate results from usability testing. This means surrounding the study participant with the fixtures, equipment, lighting, and sounds typically found in the places where the device is used. If the environment is improperly prepared, then the study results may not fully represent intended use.
Another possible source of bias for a usability study is over-training the participant. The training provided to the participant during the usability testing must be representative of what will be done in the field. Over-training the participant is considered a form of study bias. If users are typically provided an overview of the product by the manufacturer, then the usability study should simulate this. If training is not typically provided to users in the field, then training should not be provided during the usability testing. Training given to the study participant should also be given in a manner expected of the participant’s job title or responsibilities. Training for a cath lab technician using a product may be different than training provided to a clinician using the same product. If users are not expected to use the medical device immediately after training, then a training decay must also be implemented during the study. Typically, participant must have a minimum training decay period of one hour; however, more may be warranted depending on the situation. The panel stressed that this is one area where people inexperienced in human factors engineering and usability testing may unknowingly introduce inappropriate bias into a study. Not allocating a training decay of at least one hour prior to the study can invalidate the usability testing for the Human Factors submission. So, be sure to provide representative training to participants during your usability study.
If you, or your organization would like further information on how to properly conduct usability testing, we are happy to help. UserWise is experienced at conducting usability studies and successful Human Factors Submissions and is prepared to assist you with any questions. Reach out to us on our contact page.
︎Garrett Murray & Denise Forkey | December 20, 2017
Related Posts
UserWise, LLC
Office Hours
Monday-Friday 9:00 AM - 5:00 PM PT
Copyright © 2023 UserWise, LLC. All rights reserved.