Explanations for Overall Program Ratings in Criteria Matrix
|Fidelity of Interventions||Other plausible threats to validity|
|Sampling Strategy & Implementation||Replications|
|Measures||Cultural & Age Appropriateness|
degree to which the project findings are based in clear and
1 = no information about theory or hypotheses specified
2 = very little information about theory and hypotheses specified
3 = adequate information about theory and hypotheses specified
4 = nearly complete information about theory and hypotheses specified
5 = full and complete information about theory and hypotheses specified
Fidelity of interventions--the degree to which there is clear evidence of high fidelity implementation, which may include dosage data.
1 = no or very weak evidence that most treatment participants received the full intervention
2 = weak evidence that most treatment participants received the full intervention
3 = some evidence that most treatment participants received the full intervention
4 = strong evidence that most treatment participants received the full intervention
5 = very strong evidence that nearly all treatment participants received the full intervention
Sampling strategy and implementation--the quality of sampling design and implementation.
1 = no control group; unspecified sample size or inadequate sample size
2 = inappropriate control group included or no attempt at random assignment; inadequate sample size
3 = inappropriate control group included or no attempt at random assignment; adequate sample size
4 = control group included; random assignment at individual or other level (e.g., school); adequate sample size
5 = control group included; random assignment at individual or other level (e.g., school); more than adequate sample size
Attrition--evidence of sample quality based on information about attrition.
1 = no data on attrition or very high attrition
2 = high attrition
3 = moderate attrition
4 = acceptable retention
5 = high retention
Measures--the operational relevance and psychometric quality of measures used in the evaluation, and the quality of supporting evidence.
1 = no or insufficient information about measures
2 = poor choice of measures; low psychometric qualities
3 = adequate choice of measures; mixed quality
4 = relevant measures with good psychometric qualities
5 = highly relevant measures with excellent psychometric qualities
Missing Data--the quality of implementation of data collection (e.g., amount of missing data).
1 = high quantity of missing data
2 = somewhat high quantity of missing data
3 = average amount of missing data
4 = some missing data
5 = no or almost no missing data
Data Collection--way data collected in terms of bias or demand characteristics and haphazard manner.
1 = very biased manner of data collection with high demand characteristics; data collected in haphazard manner without any standardization
2 = somewhat biased manner of data collection with some demand characteristics; data collected in haphazard manner without any standardization
3 = relatively unbiased manner of data collection; standardized method of data collection
4 = anonymous or confidentiality ensured in data collection; standardized method of data collection
5 = anonymous or confidentiality ensured in data collection; standardized method of data collection; ethnic group or gender match between data collectors and participants specified
Analysis--the appropriateness and technical adequacy of techniques of analysis, primarily statistical.
1 = no analyses reported; all analyses inappropriate or do not account for important factors
2 = some but not all analyses inappropriate or left out important factors
3 = mixed in terms of appropriateness and technical adequacy
4 = appropriate analyses but not cutting edge techniques
5 = proper, state-of-the-art analyses conducted
Other plausible threats to validity (excluding attrition)--the degree to which the evaluation design and implementation addresses and eliminates plausible alternative hypotheses concerning program effects. The degree to which the study design and implementation warrants strong causal attributions concerning program effects.
1 = high threat to validity or no ability to attribute program effects
2 = threat to validity and difficult to attribute program effects
3 = somewhat of threat to validity and mixed ability to attribute effects to the program
4 = low threat to validity and ability to attribute effects to the program
5 = no or very low threat to validity and high ability to attribute effects to the program
Replications--the exact or conceptual reproduction of both the intervention implementation and evaluation.
1 = no replication.
2 = one self-replication.
3 = two or more self-replications.
4 = one or two replications by independent evaluators.
5 = three or more replications by independent evaluators producing similar results.
Dissemination Capability--program materials developed including training in program implementation, technical assistance, standardized curriculum and evaluation materials, manuals, fidelity instrumentation, videos, recruitment forms, etc.
1 = Materials, training and technical assistance not available; in case of model that requires no curriculum (i.e., therapeutic models), training/qualified trainers and technical assistance not available.
2 = Materials available but of low quality or very limited in scope; training/qualified trainers and technical assistance either not available or limited.
3 = Materials of sufficient quality with limited technical assistance and/or training/qualified trainers.
4 = High quality materials, limited technical assistance and/or training/qualified trainers or vise versa.
5 = High quality materials, technical assistance readily available and training/qualified trainers readily available.
Cultural and Age Appropriateness.
1 = no claim of culturally or age appropriate materials targeted for specific populations.
2 = claim of cultural or age appropriate materials but no of validation.
3 = age specific but not culturally appropriate or vise versa with some face validation.
4 = some materials validation materials presented.
5 = specialized materials, culturally and age appropriate, developed and evaluated or existing validated materials targeting population used.
Integrity-- the overall level of confidence that the reviewer can place in project findings based on research design and implementation.
1 = no confidence
2 = weak, at best some confidence in results
3 = mixed, some weak, some strong characteristics
4 = strong, fairly good confidence in results
5 = high confidence in results, findings fully defensible
Utility--the overall usefulness of project findings for informing prevention theory and practice. This rating is anchored according to the following categories, and combines the strength of findings and the strength of evaluation.
1 = The evaluation produced clear findings of null or negative effects for a program with
2 = The evaluation produced findings that were predominately null or negative, though not uniform or definitive.
3 = The evaluation produced ambiguous findings because of inconsistency in result or methods weaknesses that do not provide a strong basis for programmatic or theoretical contributions.
4 = The evaluation produced positive findings that demonstrate the efficacy of the program in some areas, or support the efficacy of some components of the program.
5 = The evaluation produced clear findings supporting the efficacy of well articulated theory and program design, the study provides support for the program as a replication model.