Objective For health information technology tools to fully inform evidence-based decisions, recommendations must be reliably assessed for quality and strength of evidence. We aimed to create an annotation framework for grading recommendations regarding appropriate use of diagnostic imaging examinations.#N##N#Methods The annotation framework was created by an expert panel (clinicians in three medical specialties, medical librarians, and biomedical scientists) who developed a process for achieving consensus in assessing recommendations, and evaluated by measuring agreement in grading the strength of evidence for 120 empirically selected recommendations using the Oxford Levels of Evidence.#N##N#Results Eighty-two percent of recommendations were assigned to Level 5 (expert opinion). Inter-annotator agreement was 0.70 on initial grading (κ = 0.35, 95% CI, 0.23-0.48). After systematic discussion utilizing the annotation framework, agreement increased significantly to 0.97 (κ = 0.88, 95% CI, 0.77-0.99).#N##N#Conclusions A novel annotation framework was effective for grading the strength of evidence supporting appropriate use criteria for diagnostic imaging exams.
['Ronilda Lacson', 'Ali S. Raja', 'David Osterbur', 'Ivan K. Ip', 'Louise I. Schneider', 'Paul A. Bain', 'Carol Mita', 'Julia S. Whelan', 'Patricia C. Silveira', 'David Dement', 'Ramin Khorasani']
Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations.