Incorporating random sampling in systematic literature reviews has never been more important. In this article, we discuss major benefits of this best practice.
What is the purpose of sampling?
Sampling is the process of generating a subset of reference articles from the overall dataset that your review team will be reviewing. It is considered a best practice for a number of reasons.
The first reason is that quality can only be designed from the beginning. In order for you to ensure that your review team understands and applies the eligibility criteria correctly, it is helpful to ask everyone to review the same sample of articles.
After your reviewers complete reviewing the sample, it makes sense to turn off blinding to check concordance. If you notice conflicts early on, then the interpretation or understanding of the criteria differs among team members.
Why the need for randomization?
How you generate the sample also matters. You want to select a cross section of studies that represent the diversity that may be found in the dataset. This is where randomization becomes important. A random sample ensures that the review team is tested against a diverse sampling of references.
Random Sampling and Artificial Intelligence
The same concepts hold true when one of the reviewers is artificial intelligence. Whether it is a predictions classifier or large language model performing the task, the reviewed sample can be the basis for evaluating the model’s performance early on.
Using randomization helps to ensure that a more diverse set of articles is selected. This ensures that the AI model’s output is tested against the diversity of references.
Dividing the Work Using Random Samples
The next benefit to random sampling is to divide the work equitably among reviewers while simultaneously avoiding the introduction of bias that could stem from a division of labor based on a single attribute. This requires being able to create a set of random samples for the purposes of dividing the work.
Incorporating These Best Practices Using Rayyan
Rayyan provides the ability to create a single sample or sets of samples using several methods, including randomization. Rayyan also offers several AI tools to assist with reviewing references from predictions classifiers to semantic models to large language models and AI agents. Rayyan’s tools work in combination with one another providing very powerful and precise capabilities offering enhanced integrity at greater speed with maximum convenience.
For further reading on this topic, check these Help Center articles:
How to use the Data Sampling feature for Streamlined Collaborative Screening
Organize Your Review Team Using Rayyan to Divide the Work and Assign References
