June 2009

Document Type


Degree Name



Dept. of Medical Informatics and Clinical Epidemiology


Oregon Health & Science University


There has been a substantial growth in the number of images being created every day in healthcare settings. Effective image annotation and retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well-researched, they are limited by the quality and availability of the annotations associated with the images. Advances in techniques in computer vision have led to methods for using the image itself as the search entity. The goal of our project was to create an image retrieval system a set of 1500 upper endoscopic images from the Clinical Outcomes Research Initiative Collection. We have created a web-based multimodal image retrieval system written using the Ruby on Rails framework. Ferret, a ruby port of Lucene was used for the text indexing of the annotations for the text-based retrieval. Our database also contains a number of visual features created using image processing algorithms that allows users to perform content-based retrieval. When operating in a “query-by-example” mode, our system retrieves an ordered set of images from the test collection that are “similar” in visual content to the image being queried. We also evaluated the performance of a variety of image features and machine learning classifiers that can be used to automatically annotate the image with an image class consisting of one of eight findings. We developed a hybrid algorithm for image classification that showed improved performance compared to commonly-used classification algorithms. This enabled us to provided text-based querying capability where search words from a controlled vocabulary retrieve a set of pre-classified and annotated images matching the search criteria. Our intention was to enable users to query using either a sample image, keywords or desired image class to retrieve “similar images” from the system, along with a display of the associated information from these images. Although CBIR has great potential in patient care, research and education, purely content-based image retrieval can be quite challenging for clinical purposes due to the semantic gap. Low level global features like color and texture may not be sufficient for classification of findings. However, combining visual and textual information can greatly improve retrieval performance. Additionally, the use of distance metric learning and relevance feedback can help the system produce results that are more relevant to the user.




School of Medicine



To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.