Keywords
systematic review methodology, critical appraisal, education
Description of Proposal
Graduate students in education at my university conduct systematic reviews (SRs) as part of their dissertations as well as for publications. During the initial consultation with a graduate student, I often review the steps of a SR and suggest resources about the SR methodology. Yet, students seem to prefer to utilize journals articles as a guide, even though these example articles might not report all the SR steps. For example, limiting a search to peer reviewed journals is frequently used as a quality indicator, despite critical appraisal or quality assessment of the included articles being one of the key steps in conducting a SR.
This poster will present the findings of a study examining the SRs published in Review of Educational Research to determine what sources for SR methodology are being cited, whether critical appraisal or quality assessment is mentioned, and what tools were used for critical appraisal. Review of Educational Research was chosen due to its prominence in the education discipline. In September 2021, I conducted a search within Review of Educational Research via the SAGE platform for “systematic review*” OR “systematic literature review*”. The search was limited to 2000 to present, as the Campbell Collaboration was established in 2000, and retrieved 131 studies. Due to the focus on systematic review methods, to be included, the title or abstract of the article had to state the phrase “systematic review” or “systematic literature review.”
Forty-seven SRs were included. These reviews were published between 2010 and 2021. Thirty-five articles (74%) included at least one citation related to SR methodology. PRISMA was the systematic review guidelines cited most often, followed by Systematic Reviews in the Social Sciences by Petticrew and Roberts (2006). Only 17 articles (36%) explicitly described the use of critical appraisal as a step in the systematic review process. Almost half of the articles (23, 49%) did not mention critical appraisal at all. The first explicit use of critical appraisal was in 2014 and it has been consistently used since. However, recent reviews from 2019, 2020, and 2021 have been published without mention of critical appraisal. The most popular tools used for critical appraisal were the CASP (Critical Appraisal Skills Programme) Checklists, followed by references to the Cochrane Collaboration.
The results of this study provide evidence to show education graduate students the most popular SR methodological works and the trends in the use of critical appraisal. The PRISMA statement, Systematic Reviews in the Social Sciences, and the CASP Checklists should be recommended in consultations, added to LibGuides, and included in training resources and highlighted as the resources that are most frequently cited in Review of Educational Research. Recommending resources used by scholars in a prominent education journal will offer evidence to support librarian recommendations in order to enhance the quality graduate students’ SRs.
Trends in Review of Educational Research Systematic Reviews and Implications for Supporting Graduate Students
Graduate students in education at my university conduct systematic reviews (SRs) as part of their dissertations as well as for publications. During the initial consultation with a graduate student, I often review the steps of a SR and suggest resources about the SR methodology. Yet, students seem to prefer to utilize journals articles as a guide, even though these example articles might not report all the SR steps. For example, limiting a search to peer reviewed journals is frequently used as a quality indicator, despite critical appraisal or quality assessment of the included articles being one of the key steps in conducting a SR.
This poster will present the findings of a study examining the SRs published in Review of Educational Research to determine what sources for SR methodology are being cited, whether critical appraisal or quality assessment is mentioned, and what tools were used for critical appraisal. Review of Educational Research was chosen due to its prominence in the education discipline. In September 2021, I conducted a search within Review of Educational Research via the SAGE platform for “systematic review*” OR “systematic literature review*”. The search was limited to 2000 to present, as the Campbell Collaboration was established in 2000, and retrieved 131 studies. Due to the focus on systematic review methods, to be included, the title or abstract of the article had to state the phrase “systematic review” or “systematic literature review.”
Forty-seven SRs were included. These reviews were published between 2010 and 2021. Thirty-five articles (74%) included at least one citation related to SR methodology. PRISMA was the systematic review guidelines cited most often, followed by Systematic Reviews in the Social Sciences by Petticrew and Roberts (2006). Only 17 articles (36%) explicitly described the use of critical appraisal as a step in the systematic review process. Almost half of the articles (23, 49%) did not mention critical appraisal at all. The first explicit use of critical appraisal was in 2014 and it has been consistently used since. However, recent reviews from 2019, 2020, and 2021 have been published without mention of critical appraisal. The most popular tools used for critical appraisal were the CASP (Critical Appraisal Skills Programme) Checklists, followed by references to the Cochrane Collaboration.
The results of this study provide evidence to show education graduate students the most popular SR methodological works and the trends in the use of critical appraisal. The PRISMA statement, Systematic Reviews in the Social Sciences, and the CASP Checklists should be recommended in consultations, added to LibGuides, and included in training resources and highlighted as the resources that are most frequently cited in Review of Educational Research. Recommending resources used by scholars in a prominent education journal will offer evidence to support librarian recommendations in order to enhance the quality graduate students’ SRs.
What takeaways will attendees learn from your session?