Strengthening Americas Families HOME Got to Program ListGo to Program MatrixGo to Rating CriteriaGo to Matrix ClassificationGo to Program Types 

Explanations for Overall Program Ratings in Criteria Matrix
Strengthening America's Families Project

October, 1999

Reviewers independently rate the 13 dimensions listed below and then are required to come to consensus regarding the quality of the program in question. Programs rated as exemplary programs are those that are well-implemented, are rigorously evaluated, and have consistent positive findings (integrity ratings of "A4 " or "A5 "). Model programs are those that have consistent integrity ratings of "A3" and "A4" and promising programs are those that have mixed integrity ratings but demonstrate high integrity ratings in at least 3 - 4 categories.

The ratings for the 14 dimensions below range from A1 for "very low quality," to A5 for "very high quality," with detailed criteria following each dimension.

  Theory   cube   Analysis
  Fidelity of Interventions   cube   Other plausible threats to validity
  Sampling Strategy & Implementation   cube   Replications
  Attrition   cube   Dissemination Capability
  Measures   cube   Cultural & Age Appropriateness
  Missing Data   cube   Integrity
  Data Collection   cube   Utility

1   Theory--the degree to which the project findings are based in clear and well-articulated theory, clearly stated hypotheses, and clear operational relevance.

1 = no information about theory or hypotheses specified

2 = very little information about theory and hypotheses specified

3 = adequate information about theory and hypotheses specified

4 = nearly complete information about theory and hypotheses specified

5 = full and complete information about theory and hypotheses specified

2   Fidelity of interventions--the degree to which there is clear evidence of high fidelity implementation, which may include dosage data.

1 = no or very weak evidence that most treatment participants received the full intervention

2 = weak evidence that most treatment participants received the full intervention

3 = some evidence that most treatment participants received the full intervention

4 = strong evidence that most treatment participants received the full intervention

5 = very strong evidence that nearly all treatment participants received the full intervention

3   Sampling strategy and implementation--the quality of sampling design and implementation.

1 = no control group; unspecified sample size or inadequate sample size

2 = inappropriate control group included or no attempt at random assignment; inadequate sample size

3 = inappropriate control group included or no attempt at random assignment; adequate sample size

4 = control group included; random assignment at individual or other level (e.g., school); adequate sample size

5 = control group included; random assignment at individual or other level (e.g., school); more than adequate sample size

4   Attrition--evidence of sample quality based on information about attrition.

1 = no data on attrition or very high attrition

2 = high attrition

3 = moderate attrition

4 = acceptable retention

5 = high retention

5   Measures--the operational relevance and psychometric quality of measures used in the evaluation, and the quality of supporting evidence.

1 = no or insufficient information about measures

2 = poor choice of measures; low psychometric qualities

3 = adequate choice of measures; mixed quality

4 = relevant measures with good psychometric qualities

5 = highly relevant measures with excellent psychometric qualities

6   Missing Data--the quality of implementation of data collection (e.g., amount of missing data).

1 = high quantity of missing data

2 = somewhat high quantity of missing data

3 = average amount of missing data

4 = some missing data

5 = no or almost no missing data

7   Data Collection--way data collected in terms of bias or demand characteristics and haphazard manner.

1 = very biased manner of data collection with high demand characteristics; data collected in haphazard manner without any standardization

2 = somewhat biased manner of data collection with some demand characteristics; data collected in haphazard manner without any standardization

3 = relatively unbiased manner of data collection; standardized method of data collection

4 = anonymous or confidentiality ensured in data collection; standardized method of data collection

5 = anonymous or confidentiality ensured in data collection; standardized method of data collection; ethnic group or gender match between data collectors and participants specified

8   Analysis--the appropriateness and technical adequacy of techniques of analysis, primarily statistical.

1 = no analyses reported; all analyses inappropriate or do not account for important factors

2 = some but not all analyses inappropriate or left out important factors

3 = mixed in terms of appropriateness and technical adequacy

4 = appropriate analyses but not cutting edge techniques

5 = proper, state-of-the-art analyses conducted

9   Other plausible threats to validity (excluding attrition)--the degree to which the evaluation design and implementation addresses and eliminates plausible alternative hypotheses concerning program effects. The degree to which the study design and implementation warrants strong causal attributions concerning program effects.

1 = high threat to validity or no ability to attribute program effects

2 = threat to validity and difficult to attribute program effects

3 = somewhat of threat to validity and mixed ability to attribute effects to the program

4 = low threat to validity and ability to attribute effects to the program

5 = no or very low threat to validity and high ability to attribute effects to the program

10   Replications--the exact or conceptual reproduction of both the intervention implementation and evaluation.

1 = no replication.

2 = one self-replication.

3 = two or more self-replications.

4 = one or two replications by independent evaluators.

5 = three or more replications by independent evaluators producing similar results.

11   Dissemination Capability--program materials developed including training in program implementation, technical assistance, standardized curriculum and evaluation materials, manuals, fidelity instrumentation, videos, recruitment forms, etc.

1 = Materials, training and technical assistance not available; in case of model that requires no curriculum (i.e., therapeutic models), training/qualified trainers and technical assistance not available.

2 = Materials available but of low quality or very limited in scope; training/qualified trainers and technical assistance either not available or limited.

3 = Materials of sufficient quality with limited technical assistance and/or training/qualified trainers.

4 = High quality materials, limited technical assistance and/or training/qualified trainers or vise versa.

5 = High quality materials, technical assistance readily available and training/qualified trainers readily available.

12   Cultural and Age Appropriateness.

1 = no claim of culturally or age appropriate materials targeted for specific populations.

2 = claim of cultural or age appropriate materials but no of validation.

3 = age specific but not culturally appropriate or vise versa with some face validation.

4 = some materials validation materials presented.

5 = specialized materials, culturally and age appropriate, developed and evaluated or existing validated materials targeting population used.

13   Integrity-- the overall level of confidence that the reviewer can place in project findings based on research design and implementation.

1 = no confidence

2 = weak, at best some confidence in results

3 = mixed, some weak, some strong characteristics

4 = strong, fairly good confidence in results

5 = high confidence in results, findings fully defensible

14   Utility--the overall usefulness of project findings for informing prevention theory and practice. This rating is anchored according to the following categories, and combines the strength of findings and the strength of evaluation.

1 = The evaluation produced clear findings of null or negative effects for a program with well-articulated theory and program design, the study provides support for rejecting the program as a replication model.

2 = The evaluation produced findings that were predominately null or negative, though not uniform or definitive.

3 = The evaluation produced ambiguous findings because of inconsistency in result or methods weaknesses that do not provide a strong basis for programmatic or theoretical contributions.

4 = The evaluation produced positive findings that demonstrate the efficacy of the program in some areas, or support the efficacy of some components of the program.

5 = The evaluation produced clear findings supporting the efficacy of well articulated theory and program design, the study provides support for the program as a replication model.



About   |   Literature Review   |   Model Programs   |   Helpful Links

Dept. of Health Promotion and Education