Was the design experimental, quasi-experimental, or nonexperimental? What specific design was used? Was this a cause-probing study? Given the type of question (Therapy, Prognosis, etc.), was the most rigorous possible design used?
What type of comparison was called for in the research design? Was the comparison strategy effective in illuminating key relationships?
If the study involved an intervention, were the intervention and control conditions adequately described? Was blinding used, and if so, who was blinded? If not, is there a good rationale for failure to use blinding?
If the study was nonexperimental, why did the researcher opt not to intervene? If the study was cause-probing, which criteria for inferring causality were potentially compromised? Was a retrospective or prospective design used, and was such a design appropriate?
Was the study longitudinal or cross-sectional? Was the number and timing of data collection points appropriate?
What did the researcher do to control confounding participant characteristics, and were the procedures effective? What are the threats to the study’s internal validity? Did the design enable the researcher to draw causal inferences about the relationship between the independent variable and the outcome?
What are the major limitations of the design used? Were these limitations acknowledged by the researcher and taken into account in interpreting results? What can be said about the study’s external validity?
Did the report describe an explicit theoretical or conceptual framework for the study? If not, does the absence of a framework detract from the study’s conceptual integration?
Did the report adequately describe the major features of the theory or model so that readers could understand the conceptual basis of the study?
Is the theory or model appropriate for the research problem? Does the purported link between the problem and the framework seem contrived?
Was the theory or model used for generating hypotheses, or is it used as an organizational or interpretive framework? Do the hypotheses (if any) naturally flow from the framework?