Dept. of Medical Informatics & Clinical Epidemiology
Oregon Health & Science University
Benchmarking protocols have long been used to measure the performance of both hardware and software products, although rigorous, quantitative comparisons of electronic health record (EHR) solutions are rare in the healthcare informatics marketplace or the research arena. This study uses a prototype software application, EHR Test Bench, to benchmark the performance of EHR software systems. The benchmarking program prompts a test subject to view a video of a simulated patient visit and then measures the time taken to create an electronic record of the visit using a commercial EHR charting system. The videos employ volunteer actors to play the roles of patient and doctor, and were scripted to illustrate simple clinical scenarios common in emergency department practice. The study sought to quantify two key metrics for each of the scenarios: charting efficiency and charting accuracy. The charting efficiency was defined as running time of the video divided by the total time required to watch the video and finish a completed chart, with a theoretical maximum of 100% (user finishing chart when video finishes). Documentation accuracy was measured by scoring each chart against a standard template of history and exam findings predetermined for each case, with a maximum of 100% which indicates complete capture of all data points. A total of 10 ED clinicians have taken part in the study to date. Baseline characteristics of the study population were surveyed---users averaged 10.7 years of experience with the EHR system being benchmarked and 100% rated themselves as intermediate or advanced users. Adequate inter-observer reliability among testers was noted on calibration cycles (Cronbach’s alpha = .73). Each subject watched three case videos during the session. Charting efficiency varied among users and cases: the first case (ankle sprain) was 37.5% (S.D. ± 11.8%), the second case (finger laceration) was 39.3% (S.D. ± 10.0%), and the final case (appendicitis) was 42.4 (S.D. ± 11.9%). Charting accuracy was likewise variable, ranging from 83% (S.D. ± 9%) for the first to 78% (S.D. ± 8%) for the second to 67% (S.D. ± 19%) for the last case. The small sample size at this point did not permit analysis of any correlation between charting speed and accuracy, or the factors such as provider type (physician versus physician assistant) and typing skills in benchmarking parameters. A software evaluation survey of the application completed after the testing protocol was favorable, with over 50% willing to participate in further studies. This preliminary report suggests that rigorous, quantitative benchmarking of EHR systems can be performed by end users using inexpensive and easily created study materials. Future studies of this type are urgently needed. Web-based testing, using much larger sample sizes and employing multiple different EHR systems, will allow for more statistically powerful and clinically relevant benchmark testing.
School of Medicine
Baker, Paul B., "EHR Test Bench : a Prototype Application for Quantitative Benchmarking of Electronic Health Record Software" (2014). Scholar Archive. 3496.