Dynamics of relevance judgment during physicians’ online information search process for patient treatment
Main Article Content
Abstract
Physicians rely on online sources of information more so than ever, to find the information they need when treating patients, but little is known about determining the relevance criteria when searching for online sources. Relevance judgment criteria can change with a user’s background knowledge on a search topic. Physicians are a subset of users who have received insufficient attention in this relevance criteria research topic. Due to the dynamic and multidimensional nature of relevance judgments, it is important to understand how relevance judgment criteria change in the course of information search. This study aims to determine the dynamics in relevance judgment criteria in physicians’ information search process. We observed ten internal medicine physicians while they searched the online clinical information resource to resolve a clinical question concerning patient treatment. They rated on a checklist to assess relevance judgment criteria at each of the three stages in the search process, namely, problem recognition (before searching), system interaction (searching for online resources), and document interaction (selecting the final documents after searching). We conducted pre-interview and post-interview to probe physicians search behaviours and reasons for the assessments. Topical relevance, accuracy, and credibility were found to be the most important criteria for all three search stages; however, ratings of some relevance judgment criteria, such as title, obtainability, personal preference, and understandability, changed markedly across the search process. The criteria regarding the information content tended to be more important during the problem recognition stage, while the criteria in the information format became more important as the search progressed. Physicians preferred recently published review articles in highly reliable journals with well-written article outlines, which allowed them to have quick and clear understanding of the contents. By identifying specific relevance criteria prominent at each search stage, our study revealed that physicians employ a unique set of criteria when judging the relevance. Given the fact that physicians are relying on online resources to find answers to patient care, this study can help in developing the effective clinical information systems for physicians and integrating into the Electronic Health Record (EHR) systems.
Downloads
Article Details
It is a condition of publication that manuscripts submitted to the journal have not been published, accepted for publication, nor simultaneously submitted for publication elsewhere. By submitting a manuscript, the author(s) agree that copyright for the article is transferred to the publisher, if and when the manuscript is accepted for publication.
References
Barry, C.L and Schamber, L. 1998. Users’ criteria for relevance evaluation: A cross-situational comparison. Information Processing & Management, Vol. 34, no.2–3: 219–36.
Bateman, J. 1998. Changes in relevance criteria: A longitudinal study. Proceedings of the 61st American Society for Information Science Annual Meeting (ASIS’ 98), Vol. 8, no. 23-32.
Bruce, H.W. 1994. A cognitive view of the situational dynamism of user-centered relevance estimation. Journal of the American Society for Information Science, Vol. 45, No. 3: 142–148.
Clarke, M.A., Belden, J,L,, Koopman, R.J., Steege, L.M., Moore, J.L., Canfield, S.M. and Kim, M.S. 2013. Information needs and information-seeking behaviour analysis of primary care physicians and nurses: a literature review. Health Information & Libraries Journal, Vol.30, no.3: 178–90.
Daei, A., Soleymani, M.R., Ashrafi-rizi, H., Zargham-Boroujeni, A. and Kelishadi, R. 2020a. Clinical information seeking behavior of physicians: A systematic review. International Journal of Medical Informatics. Available at: https://doi.org/10.1016/ j.ijmedinf.2020.104144.
Daei, A., Soleymani, M. R., Ashrafi‐rizi, H., Kelishadi, R., and Zargham‑Boroujeni, A. 2020b. Personal, technical and organisational factors affect whether physicians seek answers to clinical questions during patient care: A literature review. Health Information & Libraries Journal. Available at: https://doi.org/10.1111/hir.12323.
Davies, K. 2011. Information needs and barriers to accessing electronic information: hospital-based physicians compared to primary care physicians. Journal of Hospital Librarianship, Vol.11, no.3: 249–60.
Del Fiol, G., Workman, T.E. and Gorman, P.N. 2014. Clinical questions raised by clinicians at the point of care: a systematic review. JAMA Internal Medicine, Vol. 174, no. 5: 710–718.
Ely, J., Sheroff, J., Hambliss, M., Bell, M. and Osenbaum M. 2005. Answering physicians’ clinical questions: Obstacles and potential solutions. Journal of the American Medical Informatics Association, Vol. 12, no. 2: 217–24.
Hollis, K. F., Roberts, K., Bedrick, S., and Hersh, W. R. 2020. Addressing the search challenges of precision medicine with information retrieval systems and physician readers. 30th Medical Informatics Europe Conference (MIE 2020) (pp. 813-817). IOS Press.
Grad, R., Pluye, P., Granikov, V., Johnson-lafleur, J., Shulha, M., Sridhar, S.B., Moscovici, J.L., Bartlett, G., Vandal, A.C., Marlow, B. and Kloda, L. 2011. Physicians’ assessment of the value of clinical information: operationalization of a theoretical model. Journal of the American Society for Information Science and Technology, Vol. 62, no. 10: 1884–1891.
Hersh, W. 1994. Relevance and retrieval evaluation: Perspectives from medicine. Journal of the American Society for Information Science, Vol. 45, no.3: 201-206.
Hughes, B., Wareham, J. and Joshi, I. 2010. Doctors' online information needs, cognitive search strategies, and judgments of information quality and cognitive authority: How predictive judgments introduce bias into cognitive search models. Journal of the American Society for Information Science and Technology, Vol. 61, no.3, 433-452.
Janes, J.W. 1991. Relevance judgments and the incremental presentation of document representations. Information Processing & Management, Vol. 27, no.6: 629–46.
Kuhlthau, C. C. 1991. Inside the search process: Information seeking from the user's perspective. Journal of the American Society for Information Science, Vol. 42, no. 5: 361-371.
Macedo-Rouet, M., Rouet, J. F., Ros, C. and Vibert, N. 2012. How do scientists select articles in the PubMed database? An empirical study of criteria and strategies. European Review of Applied Psychology, Vol. 62, no. 2: 63-72.
Maglaughlin, K.L. and Sonnenwald, D.H. 2002. User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments. Journal of the American Society for Information Science and Technology, Vol. 53, no. 5: 327–42.
Mikalef, P., Kourouthanassis, P. E. and Pateli, A. G. 2017. Online information search behaviour of physicians. Health Information & Libraries Journal, Vol. 34, no. 1: 58-73.
Pluye, P., Grad, R.M., Dunikowski, L.G. and Stephenson, R. 2005. Impact of clinical information-retrieval technology on physicians: a literature review of quantitative, qualitative and mixed methods studies. International Journal of Medical Informatics, Vol.74, no. 9: 745–68. Available at: https://doi.org/10.1016/j.ijmedinf.2005.05.004.
Qu, J., Arguello, J., and Wang, Y. 2020. Towards explainable retrieval models for precision medicine literature search. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1593-1596). Available at: https://doi.org/10.1145/3397271.3401277.
Saracevic T. 1997. The stratified model of information retrieval interaction: Extension and applications. Proceedings of the Annual Meeting-American Society for Information Science, Vol. 34: 313–27.
Saracevic. T. and Kantor, K.B. 1997. Studying the value of library and information services: Part I. Establishing a theoretical framework. Journal of the American Society for Information Science, Vol. 48, no.6: 527–542.
Schamber, L. 1991. Users' criteria for evaluation in a multimedia environment. Proceedings of the ASIS Annual Meeting, Vol.28: 126–33.
Spink, A., Greisdorf, H. and Bateman J. 1998. From highly relevant to not relevant: examining different regions of relevance. Information Processing & Management, Vol.34, no. 5: 599–621.
Tang, R. and Solomon P. 2001. Use of relevance criteria across stages of document evaluation: On the complementarity of experimental and naturalistic studies. Journal of the American Society for Information Science and Technology, Vol. 52, no. 8: 676–85.
Taylor, A.R., Cool, C., Belkin, N.J., and Amadio, W.J. 2007. Relationships between categories of relevance criteria and stage in task completion. Information Processing & Management, Vol. 43, no. 4: 1071–1084.
Uprety, S., Su, Y., Song, D. and Li, J. 2018. Modeling multidimensional user relevance in IR using vector spaces. 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 993-996). Available at: https://doi.org/10.1145/3209978.3210130
Vakkari, P., Pennanen, M. and Serola, S. 2003. Changes of search terms and tactics while writing a research proposal. Information Processing & Management, Vol. 39, no. 3: 445–63.
Vibert, N., Ros, C., Bigot, L. L., Ramond, M., Gatefin, J. and Rouet, J. F. 2009. Effects of domain knowledge on reference search with the PubMed database: An experimental study. Journal of the American Society for Information Science and Technology, Vol. 60, no. 7: 1423-1447.
Wang, P. and Soergel D. 1998. A cognitive model of document use during a research project. Study I. Document selection. Journal of the American Society for Information Science, Vol.49, no. 2: 115–33.
Wang, P.L. and White, M.D. 1999. A cognitive model of document use during a research project. Study II. Decisions at the reading and citing stages. Journal of the American Society for Information Science, Vol.50, no. 2: 98–114.
Westbrook, J.I, Gosling, A.S, and Westbrook, M.T. 2005. Use of point-of-care online clinical evidence by junior and senior doctors in New South Wales public hospitals. Internal Medicine Journal, Vol. 35, no. 7: 399–404.