Critical Appraisal – Why and How
Evidence Based Medicine
Healthcare and healthcare research are comprised of several steps and these steps fall under the umbrella of evidence based medicine (EBM). According to a 2006 JAMA systematic review: “Evidence based medicine is an approach of practicing medicine with the goal to improve and evaluate patient care.” Included in this process are what’s commonly called the “EBM Cycle” or “Steps to EBM.” These steps are: ASK, ACQUIRE, APPRAISE, and APPLY.
We have several resources on the EBM process and librarians are available to review this with students, residents, and faculty at any time. Recently, we presented an introduction to critical appraisal which is included in the “Appraise” step of the EBM Cycle. Critical appraisal is an extremely important, but often overlooked, part of evidence based medicine practices and we are here to help make this process easier.
Why is critical appraisal important?
In very general terms, critical appraisal is important because not everything that’s been published, even “peer-reviewed” is of good quality or appropriate for your needs. This is especially relevant in the age of paper mills and AI when research articles are being churned out an ever-increasing rate. Further, critical appraisal is directly related to critical thinking which is what separates humans from machines (i.e., AI). What does this mean? This means that healthcare professionals must:
- Question
- Test previously held assumptions
- Acknowledge ambiguity
- Interpret
- Reflect
- Solve problems
What is critical appraisal?
According to Morris, Fewell, and Oleszewski in 2012, critical appraisal is the process of carefully and systematically reviewing the evidence. Statistician Richard Royall of Johns Hopkins Bloomberg School of Public Health, said that there are three questions a scientist might want to ask to appraise a study:
- ‘What is the evidence?’
- ‘What should I believe?’ and
- ‘What should I do?’
Six Steps to Critical Appraisal
The 2012 Morris article breaks this down further by describing the following steps researchers and practitioners should follow:
1.) Analyze the question the study is examining: Determine what is being studied – what question is the author trying to answer?
2.) Analyze the study design: What is the study design? Was the appropriate study type chosen for the research question identified? Remember: the question being asked will influence the type of study – different hierarchies will apply depending on whether the question is prognosis, diagnosis, treatment, or harm. Review the CEBM Levels of Evidence Table for examples of appropriate studies/research for various questions.
3.) Analyze the evidence presented (consider the Pyramid of Evidence): Is the quality of evidence provided by the study good enough to support the study’s recommendations? “The initial creation of an evidence hierarchy was intended to link the quality of evidence to the soundness of the recommendations.” The hierarchy was created to “favor study designs that are less susceptible to selection or treatment bias.” (Morris 2012)
4.) Determine if the information is applicable to patient care: Does this observed difference affect the approach to treatment or diagnosis for patients? How big is the result? Is it big enough? Is it meaningful? Is it doable?
5.) Look for bias: Who is left out of the study and why? Did the type of study used result in selection or treatment bias? Visit the Catalogue of Bias for more details on recognizing bias.
6.) Analyze the statistics: Are the numbers portrayed in an accurate manner and in a way that they can be easily replicated?
Tools you can use to help critically appraise research
- Critical Appraisal: Assessing Risk of Bias – Evidence Based Medicine – Research Guides at Dartmouth College
- CASP Checklists – Critical Appraisal Skills Programme
- Critical Appraisal tools — Centre for Evidence-Based Medicine (CEBM), University of Oxford
- JBI Critical Appraisal Tools | JBI
Further Reading
Guyatt, G., Rennie, R., Meade, M., & Cook, D. (2015). User’s guide to the medical literature: A manual for evidence-based clinical practice. (3rd ed.) Chicago, IL: Medical Library Association.
NOTE: JAMA Evidence also provides access to audio discussions by the authors to accompany many chapters. If you prefer auditory learning, these are a good alternative to reading the chapters.
McAlister FA, Straus SE, Guyatt GH, Haynes RB, for the Evidence-Based Medicine Working Group. Users’ Guides to the Medical Literature: XX. Integrating Research Evidence With the Care of the Individual Patient. JAMA. 2000;283(21):2829–2836. doi:10.1001/jama.283.21.2829
Morris, M.J., Fewell, A.E,. Oleszewski, R.T. Evidence-Based Medicine: Specific Skills Necessary for Developing Expertise in Critical Appraisal. Southern Medical Journal. 105 (3) 2012.
Saldanha IJ, Skelly AC, Ley KV, et al. Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2022 Sep. Table 1, Common types of nonrandomized studies of interventions (NRSIs) based on study design features. Available from: https://www.ncbi.nlm.nih.gov/books/NBK584468/table/ch4.tab1/
Shaneyfelt T, Baum KD, Bell D, et al. Instruments for Evaluating Education in Evidence-Based Practice: A Systematic Review. JAMA. 2006;296(9):1116–1127. doi:10.1001/jama.296.9.1116
Windish, D. (2013) Searching for the right evidence: how to answer your clinical questions using the 6S hierarchy. Evidence-Based Medicine, 18(3): 93-97.
Research Guides
- Additional Reading – Evidence Based Health Care – Research Guides at University of North Dakota
- Appraise – Social Work – Research Guides at University of North Dakota
- Critical Appraisal Tools & Calculators – Evidence Based Health Care – Research Guides at University of North Dakota
- Evidence Based Practice – Nursing – Research Guides at University of Michigan Library
- Medical Research Tutorial – NW Clinical Campus (Minot) – Research Guides at University of North Dakota
- Overview of EBP – Evidence-Based Practice – Guides at Medical University of South Carolina
- Reading List – Jump into Evidence Based Practice – Research Guides at University of North Dakota
