Everyone Wants to do a Systematic Review

It isn’t your imagination, more and more people are writing and publishing systematic reviews. In a recent research letter from JAMA Internal Medicine, Assessment of Publication Trends of Systematic Reviews and Randomized Clinical Trials, 1995 to 2017, the authors noted the rate of growth in published systematic reviews was ginormous. I know ginormous is not a technical term and the authors would not use it in a publication like JAMA Internal Medicine, but when the rate of growth is 4676% I think the word ginormous is appropriate.

The systematic review is the new little black dress on the publishing runway, everybody must have one dahling.

But not all systematic reviews are created equal. The JAMA Internal Medicine study did not look at the quality of these systematic reviews. It just looked at the number of publications across 18 medical and surgical areas that made it into PubMed and were indexed as systematic reviews. Many systematic reviews do not meet basic standards. (Why are we calling them systematic reviews then?!) Melissa Rethlefsen reports in the Journal of Clinical Epidemiology. 2015 Jun;68(6):17-26, systematic reviews that included librarians or an information specialist as a part of the process were more likely to be reproducible and meet specific search standards.

So, it would behoove people who want to do a systematic review to contact a librarian to help. That is a good thing. Unfortunately, it seems too many people don’t understand what is really needed to do a proper systematic review. They just want that little black dress, and they want it now.

Librarians are not only drowning in the requests for systematic reviews, but the pool is continually filling with people who refuse to understand the requirements for a systematic review, or they are delusional about the commitment in time and effort needed to do one. Librarians are continually trying to lower the water level through education, but those educational methods are not hitting home with the right people.

I have seen librarians post information sheets online detailing the process. I know some librarians require forms and “contracts” to be fully filled out prior to discussions. Yet there still are those who are surprised by search results in the thousands (even though they were informed this is typical and they need to go through them) and ask for fewer results. There are systematic review searches that the librarians spend days creating that are languishing in systems like Covidence or DistillerSR never to be revisited by the requestor.

It isn’t that librarians don’t want to do systematic reviews, they don’t want to waste their time (often several days) on something that goes nowhere or is of such poor quality they need to ask that their name be removed from the paper. So they continue with their education efforts, guiding researchers to on how to do a proper systematic review or suggesting different types of searches as appropriate.

However, I feel this approach is a little bit like the librarian sticking their finger in the leaking damn. It isn’t until somebody repairs the damn will the leak actually stop and prevent the flooding of poor systematic reviews.

Publishers, faculty, researchers, and authors need to step up and help repair the leak.

  1. Publishers need to make it very clear to authors the criteria needed for a systematic review and anything that doesn’t meet the criteria will be immediately rejected. Some publishers are good at this, many are not. We have over a 4000% increase in published systematic reviews, clearly some publishers are not that strict.
  2. Faculty need to stop assigning systematic reviews as a summer project to their students. Assigning how to learn the methods or requirements to conduct a systematic review is entirely achievable for the summer. Telling students, residents, and other junior researchers to do a systematic review over the summer is not.
  3. Researchers and authors need to have a reality self check. We are not lying when we say a proper systematic review typically takes 12-24 months. You will not be the one who is “different” and can get it done in 4-6 months. You will also need at least 3 colleagues who have same time, opportunity, and dedication as you do because you need at least 3 reviewers (including yourself) to minimalize reviewer bias.

A quality little black dress is something that you will come back to and use as often as the need arises. A good systematic review is something that people can utilize to form policies, treat patients, and base recommendations. Unfortunately, a poorly done systematic review is worse than ill fitted, poorly stitched, little black dress made out of cheap material. Poorly done systematic reviews not only waste time but it flood the databases and do little to improve policy or treat patients. But hey, you got that little black dress published and it is now on your CV, so who cares if we pull a thread and it all falls apart. You don’t need to be the Ralph Lauren of systematic reviews, but you also shouldn’t be Amazon version either.

**Quick edit/update**
After posting this, I started thinking we librarians need to start being more assertive when somebody wants a “systematic review” without knowing or wanting to put in all of the work of actually conducting a systematic review. In addition to educating them, we have to remember to say No. It is hard to say no, librarians don’t like to say no. Perhaps we should also say no.

*Disclaimer* The authors in the JAMA Internal Medicine cited above are from my institution and one of them is my librarian co-worker.

3 thoughts on “Everyone Wants to do a Systematic Review”

  1. Thanks so much for writing this Michelle! I will be sharing it with colleagues (and possibly some faculty)
    Just yesterday, there was a story in our campus news about a systematic review I remembered being consulted about. After meeting a couple of times, I offered to review the grad students search strategy but he didn’t take me up on it.
    It must have turned out rather poorly since his initial set of articles numbered over 30,000! ( see https://peerj.com/articles/7390/?utm_source=TrendMD&utm_campaign=PeerJ_TrendMD_0&utm_medium=TrendMD#fig-1)
    Strangely, for a systematic review, the search strategy was not included as an appendix ­čÖü
    It’s very distressing to see this published in PeerJ, which is highly selective. A clear illustration of your Point 1.

  2. On your first suggestion, I wonder if there is some potential for a joint approach by publishers, journal editors and librarians. EAHIL is interested in working with any interested parties, particularly to improve reporting standards of SRs, but also to improve SR quality generally. Is this a topic where a joint approach, by European, North American, and indeed health librarians’ organisations elsewhere, might work?

Comments are closed.