July 2009

Document Type


Degree Name



Dept. of Medical Informatics and Clinical Epidemiology


Oregon Health & Science University


Objective This study evaluated the level of agreement between clinicians (experts) and nonclinicians (lay persons) when answering questions and selecting supporting text from ambulatory care encounter notes. The study hypothesized that 1) clinicians would agree more often than non-clinicians across all documents and 2) agreement would be higher for both groups when subjects were asked to find explicit text in documents than when the subjects were asked to draw inferences from the text. The study was designed to shed light on the causes of disagreement among coders of clinical documents. Methods Eight clinical experts and eight non-clinicians reviewed 58 clinical encounter notes, answered questions about the notes, highlighted text in support of answers, and provided comments about the reasoning behind the answers and/or text selections. Study subjects interacted with a web-based data collection tool that displayed the documents and collected user input. The data were analyzed using quantitative measures of agreement for question answers and selected text as well as qualitative methods for content analysis of the comments data. Results The quantitative analysis revealed support for Hypothesis #1 though not for Hypothesis #2, likely due to confounders in study design. However, the qualitative analysis provided important information about how subjects search for information within clinical records and attempt to resolve ambiguity, when present. Five general approaches emerged from the content analysis: 1) Explicit statements are best, if found, and lead to the highest agreement among subjects 2) all subjects utilize ad hoc heuristics based on the available data to reach conclusions, 3) poor temporal specificity creates ambiguity, 4) exceptions to common clinical presentations cause confusion among all codes, and 5) some ambiguity is irresolvable post hoc. Additionally all subjects in this study were able to identify relevant information in response to questions, regardless of clinical training. Finally, subjects appeared to disagree for predominantly non-clinical reasons. Conclusions The results suggest that there is significant work to do to mitigate or eliminate some of the causes of data ambiguity in clinical information systems (CIS). This work involves improving general cognition support (e.g., rendering collected data properly contextualized with other, available information), eliminating the use of secondary information (such as ICD-9 codes as proxies for problem lists) in the clinical record, and building heuristics to identify points of ambiguity for the clinician to resolve during a clinical encounter. Computational support to reduce ambiguity may help reduce the introduction and proliferation of ambiguity in the medical record, increasing the likelihood of higher inter-rater agreement among coders.




School of Medicine



To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.