Notice2023-23391
Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures, Draft Version 2.0
Primary source
Metadata and text below are from the Federal Register, a public-domain U.S. government work. Always verify the official published version before relying on it for any legal matter.
Published
October 24, 2023
Issuing agencies
Health and Human Services DepartmentChildren and Families Administration
Abstract
The Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), oversees the Title IV-E Prevention Services Clearinghouse. ACF seeks comments on proposed changes and clarifications to existing standards and procedures in the Handbook of Standards and Procedures, Version 2.0.
Full Text
<html>
<head>
<title>Federal Register, Volume 88 Issue 204 (Tuesday, October 24, 2023)</title>
</head>
<body><pre>
[Federal Register Volume 88, Number 204 (Tuesday, October 24, 2023)]
[Notices]
[Pages 73021-73029]
From the Federal Register Online via the Government Publishing Office [<a href="http://www.gpo.gov">www.gpo.gov</a>]
[FR Doc No: 2023-23391]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Title IV-E Prevention Services Clearinghouse Handbook of
Standards and Procedures, Draft Version 2.0
AGENCY: Administration for Children and Families, Department of Health
and Human Services.
ACTION: Request for public comment.
-----------------------------------------------------------------------
SUMMARY: The Administration for Children and Families (ACF), within the
U.S. Department of Health and Human Services (HHS), oversees the Title
IV-E Prevention Services Clearinghouse. ACF seeks comments on proposed
changes and clarifications to existing standards and procedures in the
Handbook of Standards and Procedures, Version 2.0.
DATES: The deadline for comments on this notice is November 24, 2023.
ADDRESSES: Interested parties may submit written questions, comments,
and supplementary documents by email to <a href="/cdn-cgi/l/email-protection#7a0a081f0c1f140e131514091f080c13191f093a1b180e1b0909151954191517"><span class="__cf_email__" data-cfemail="a2d2d0c7d4c7ccd6cbcdccd1c7d0d4cbc1c7d1e2c3c0d6c3d1d1cdc18cc1cdcf">[email protected]</span></a>
with ``Title IV-E Prevention Services Clearinghouse FRN comment'' in
the subject line. To ensure that your comments have maximum effect,
please identify clearly the section of the draft Handbook of Standards
and Procedures, Version 2.0 that your comments address.
Readers are referred to the full version of the draft Handbook of
Standards and Procedures, Version 2.0 on the Clearinghouse website
(<a href="https://preventionservices.acf.hhs.gov/resources/comment-draft-handbook">https://preventionservices.acf.hhs.gov/resources/comment-draft-handbook</a>).
SUPPLEMENTARY INFORMATION:
1.0 Background and Legislative Context
The Family First Prevention Services Act (FFPSA) was signed into
law as part of the Bipartisan Budget Act (H.R. 1892) on February 9,
2018. FFPSA amended the Social Security Act (the Act) to enable use of
Federal funds available under parts B and E of title IV of the Social
Security Act to provide enhanced support to children and families and
prevent foster care placements through the provision of evidence-based
``mental health and substance abuse prevention and treatment services,
in-home parent skill-based programs, and kinship navigator services.''
As described in the statutory language, these services and programs are
intended ``for children who are candidates for foster care or who are
pregnant or parenting foster youth and the parents or kin caregivers of
the children.'' The Act requires an independent systematic review of
evidence to designate programs and services as ``promising,''
``supported,'' and ``well-supported'' practices.
In order to meet these requirements, ACF established the Title IV-E
Prevention Services Clearinghouse (the Clearinghouse). The
Clearinghouse carries out a systematic review process implemented by
trained reviewers using consistent, transparent standards and
procedures. The Handbook of Standards and Procedures, Version 1.0
(<a href="https://preventionservices.acf.hhs.gov/review-process">https://preventionservices.acf.hhs.gov/review-process</a>) provides a
detailed description of the standards used to identify and review
programs and services for the Clearinghouse and the procedures followed
by the Clearinghouse staff. The Handbook of Standards and Procedures,
Version 1.0 was informed by public comments submitted in response to
Federal Register Notice 83 FR 29122 (<a href="https://www.federalregister.gov/documents/2018/06/22/2018-13420/decisions-related-to-the-development-of-a-clearinghouse-of-evidence-based-practices-in-accordance">https://www.federalregister.gov/documents/2018/06/22/2018-13420/decisions-related-to-the-development-of-a-clearinghouse-of-evidence-based-practices-in-accordance</a>),
consultations with research and practice experts, and the review
processes developed and used by other prominent evidence
clearinghouses.
2.0 Overview of 2021 Request for Public Comment on Title IV-E
Prevention Services Clearinghouse Handbook of Standards and Procedures,
Version 1.0
ACF solicited feedback on the Prevention Services Clearinghouse
Handbook of Standards and Procedures, Version 1.0 (subsequently
referred to as Handbook Version 1.0) through a Federal Register Notice
86 FR 37332 (<a href="https://www.federalregister.gov/documents/2021/07/15/2021-15065/title-iv-e-prevention-services-clearinghouse-handbook-of-standards-and-procedures">https://www.federalregister.gov/documents/2021/07/15/2021-15065/title-iv-e-prevention-services-clearinghouse-handbook-of-standards-and-procedures</a>) published on July 15, 2021. This comment
period was open for 30 days and closed on August 16, 2021. One hundred
four unique commenters submitted feedback, including 10 commenters from
state and local child welfare agencies. Commenters included state and
local government administrators, program and service developers,
Federal staff, researchers and evaluators, foundation and non-profit
organization staff, and other interested parties. ACF ensured the
careful review and consideration of all of the comments in developing
the draft Handbook of Standards and Procedures, Version 2.0
(subsequently referred to as Handbook Version 2.0). Comments were
considered within the context of the statutory requirements of FFSPA,
the necessity to conduct a systematic, objective, and transparent
evidence review, and resource considerations. The public comments
informed discussions with a large number of experts whose comments were
also considered in developing the proposed revisions.
Summary of Comments. Comments highlighted how the standards and
procedures specified in Handbook Version 1.0 might be revised to better
reflect the goals and requirements of the Executive Order on Advancing
Racial Equity and Support for Underserved Communities Through the
Federal Government. For example, commenters recommended prioritizing
the review of programs and services that have been implemented and/or
studied with diverse populations (Section 2.2). Commenters also
recommended engaging diverse individuals and those with lived
experience to inform the systematic review process and allowing greater
flexibility for culturally adapted programs and services. Commenters
[[Page 73022]]
recommended providing additional detail to clarify the existing
standards and procedures. For example, comments requested technical
clarification regarding the definition of an available written
protocol, manual, or other documentation (Section 2.1.2), determination
of the length of time after the end of treatment (Section 6.2.3),
determination of whether program or service or study adaptations are
substantial (Section 4.1.6), and calculations of effect size and
statistical significance (Section 5.1.0). Commenters recommended
broadening the definitions of the program or service areas (Section
2.1.2) to be more inclusive regarding the types of programs and
services that may be eligible for review. Commenters recommended
broadening the definition of eligible comparison conditions (Section
4.1.4) and making the design and execution standards (Chapter 5),
particularly those related to baseline equivalence (Section 5.7), more
flexible. Finally, commenters provided recommendations to ACF that did
not pertain to the Clearinghouse. For example, comments recommended ACF
provide further support and investment in building evidence,
particularly of programs and services designed to serve communities of
color and others disproportionally represented in the child welfare
system as well as for kinship navigator programs.
Summary of Proposed Revisions. The draft Handbook Version 2.0 aims
to be responsive to the diversity of comments received, to enhance the
transparency of the systematic review process, and to support efforts
to advance equity in accordance with the Executive Order on Advancing
Racial Equity and Support for Underserved Communities Through the
Federal Government. For example, revised program or service area
definitions (Section 2.1) are inclusive of a broader range of programs
and services, new program or service prioritization criteria have been
added to consider the child welfare relevance and diversity of
populations served (Section 2.2) with similar criteria also added for
study prioritization (Section 2.3), and the range of eligible
comparison conditions for studies has been expanded to include studies
that compare one intervention to another intervention (Section 4.1.7).
Additional clarification and guidance are now provided on program or
service and study adaptations, including new examples of how standards
are applied to culturally adapted programs and services (Sections 2.3.2
and 4.1.9). Clarification is also provided that eligible outcomes and
outcome measures may be defined differently across studies to reflect
the different ages, backgrounds, cultures, locations, and contexts of
the study participants, with examples provided (Section 4.1.8).
Formulae used in effect size and statistical significance calculations
are now provided directly in the Handbook (Chapter 6) and additional
guidance and clarification is provided on design confounds, including
clarification that studies with a single provider unit shared across
the intervention and comparison conditions are not considered a
confound (Section 5.9.3). A broader range of options is provided for
establishing baseline equivalence and low attrition randomized group
design contrasts are no longer assessed for baseline equivalence
(Section 5.7). The Handbook now provides additional information on how
the risk of harm assessment is conducted, with additional
considerations for cases where the comparison group receives another
intervention (Section 7.2.1). Further, additional clarification on how
time since the end of treatment is calculated is provided (Section
7.2.3). The Handbook now clearly specifies how any member of the public
can submit recommendations of programs or services for review or
information about studies of those recommended programs and services to
the Clearinghouse at any time (Chapters 1 and 3).
Additional Relevant Activities. The Clearinghouse also intends to
conduct additional activities to be responsive to public comments and
to support efforts to advance equity in accordance with the Executive
Order on Advancing Racial Equity and Support for Underserved
Communities Through the Federal Government. First, the Clearinghouse is
planning to display study participant characteristics on the program or
service page of the Clearinghouse website. Display of participant
characteristics is intended to promote transparency on the extent to
which diverse populations are represented in research reviewed by the
Clearinghouse. Second, the Clearinghouse plans to develop two new
reports focused on equity. These two reports are intended to provide
additional information about diverse populations included in studies of
the programs and services that have been reviewed by the Clearinghouse
and identify gaps in evidence. Third, enhanced activities are planned
for future public calls for program and service recommendations in
order to comprehensively identify culturally adapted and culturally
grounded programs and services that may be eligible for review. The
Clearinghouse plans to conduct targeted outreach to providers of
culturally adapted and culturally grounded programs and services and
community-based organizations serving diverse populations to improve
engagement. The Clearinghouse also plans to clearly communicate in
future public calls how the public, including community-based
organizations and providers of culturally adapted and culturally
grounded programs and services, can recommend programs and services and
submit relevant studies of programs and services to the Clearinghouse.
Further, the Clearinghouse plans to make future public call materials
available in both English and Spanish. Fourth, the Clearinghouse
intends to revise its author Reporting Guide to clarify recommended
reporting related to culturally adapted and culturally grounded
programs and services and the characteristics of their participants.
Fifth, the Clearinghouse intends to revise existing resources for
Clearinghouse users, such as its Frequently Asked Questions (FAQ)
website section and fact sheet resources, with person-centered design
principles to ensure information about the Clearinghouse and its
standards and procedures are accessible. Sixth, the Clearinghouse plans
to publicly post all programs and services that have been recommended
for review and will continue to explore additional ways to improve
transparency such as through data sharing.
A comprehensive list of specific revisions and clarifications to
the Clearinghouse's Standards and Procedures is provided in the
following section. Subsequent chapter and section numbers all refer to
the chapter and section numbering for the draft Handbook Version 2.0
unless the text explicitly indicates a reference to Handbook Version
1.0 chapter and section numbering.
3.0 Revisions and Clarifications to the Clearinghouse's Standards and
Procedures in the Draft Handbook Version 2.0
3.1 Introduction
The revised introduction includes a description of the
Clearinghouse website and resources available on the website. This
includes reference to the FAQ section that includes information on how
members of the public can submit a program or service recommendation
and how to provide information about studies to the Clearinghouse.
[[Page 73023]]
3.2 Chapter 1. Identify Programs and Services
Revisions clarify that all program and service recommendations are
retained for consideration, including those submitted during public
calls and ad hoc recommendations submitted to the Prevention Services
Clearinghouse inbox. Revisions also clarify that any member of the
public may submit a program or service recommendation at any time to
the Clearinghouse via email and that suggested information to include
as part of a program or service recommendation can be found on the FAQ
section of the Clearinghouse website. Additionally, this section now
indicates that all programs and services identified as potential
candidates for review will be posted on the Clearinghouse website.
3.3 Chapter 2. Prioritize and Select Programs and Services
3.3.1 Revisions and Clarifications to Program or Service Area
Definitions (Section 2.1.1)
Based on FRN feedback and consultation with experts in the fields
of mental health, substance use, parenting and parent skill-based
programs and services, kinship navigator programs, and child welfare,
the draft Handbook Version 2.0 revised and clarified the in-home
parent-skill based and substance use prevention and treatment program
or service area definitions, as noted below.
<bullet> In-home parent skill-based programs and services. The
revised definition is more flexible and now indicates that eligible
programs and services involve direct intervention with a parent or
caregiver and target parenting skills or other skills that can be
applied to where the child resides, including in the home. The revised
definition also clarifies that delivery of programs and services can
occur in the home or other settings and defines necessary content for a
program or services to be considered ``skill-based.''
Revised examples of eligible and ineligible in-home skill-based
programs and services are provided in Exhibit 2.3.
<bullet> Substance use prevention and treatment programs and
services. The revised definition clarifies that programs or services:
<bullet> targeting recovery from substance use (as well as those
targeting prevention, treatment, remediation, elimination and/or
reduction of substance use or misuse) are eligible; and
<bullet> without client-oriented substance use prevention or
treatment components, such as mass communications/media campaigns or
interventions that solely target broader community-level or policy
systems, remain not eligible.
Revised examples of eligible and not eligible programs and services
are now provided in Exhibit 2.2. Specifically, one new example
clarifies that programs or services targeting parents or caregivers
aiming to prevent substance use among children and youth are eligible.
Minor wording changes were made to the kinship navigator program or
service area definition for clarification purposes. Experts did not
suggest any changes to eligible outcomes for kinship navigator programs
and services.
No changes were made to the mental health prevention and treatment
programs and services definition. New examples of eligible and
ineligible programs and services are provided in Exhibit 2.1.
3.3.2 Clarifications to Available Protocols, Manuals, or Other
Documentation (Section 2.1.2)
To be eligible for review by the Prevention Services Clearinghouse,
programs and services must be clearly defined and replicable. To meet
this criterion, programs and services must have available written or
recorded protocols, manuals, or other documentation that describes how
to implement or administer the practice (referred to subsequently in
this notice as a ``manual'' for brevity). Revisions to this section
clarify that materials to satisfy this requirement may be presented in
a web-based format and that ``manual'' can include recorded videos or
online learning systems if these materials describe how to implement or
administer the practice. The Clearinghouse notes that, consistent with
Handbook Version 1.0, there are no language requirements for manual
eligibility.
3.3.3 Revisions and Clarifications to Program or Service Prioritization
(Section 2.2)
As of July 2023, the Prevention Services Clearinghouse has reviewed
148 programs and services. Yet there remains a high volume of
potentially eligible programs and services identified for review. As a
result, the Prevention Services Clearinghouse must continue to
prioritize programs and services for review. The draft Handbook Version
2.0 continues to highlight the prioritization of programs and services
with available evidence of eligibility and programs and services in
active use (Section 2.2). New to this section is further clarification
about additional prioritization considerations. These additional
prioritization criteria were informed by recommendations from public
comments and consultation with experts. Listed below are the additional
prioritization criteria included in the draft Handbook Version 2.0.
<bullet> Number and source of program or service recommendations
received;
<bullet> Child welfare relevance;
<bullet> Population(s) served;
<bullet> Previous evaluations and studies; and
<bullet> Implementation supports.
The Clearinghouse continues to prioritize programs and services in
a way that ensures representation across the four program and service
areas. Additional clarification is provided in draft Handbook Version
2.0 noting that the Clearinghouse assesses prioritization criteria by
examining publicly available information, other clearinghouses'
websites, and materials submitted with program or service
recommendations.
3.3.4 Clarifications on Program or Service Selection (Section 2.3.1)
Given the large volume of programs and services identified,
resource considerations mean that not all programs and services can be
selected for review at once. To help clarify the distinction between
the prioritization and reviewing process, the draft Handbook Version
2.0 adds a new section on selection of a program or service for review
(Section 2.3.1). Based on the prioritization process, specific programs
and services are selected for review at a given time, as indicated by
publication on the working list of programs and services planned for
review available on the Prevention Services Clearinghouse website. The
final eligibility of a program or service for review by the
Clearinghouse is determined after a program or service is selected for
the working list.
3.3.5 Revisions to Program or Service Adaptations Criteria (Section
2.3.2)
Multiple public comments requested clarification regarding the
program or service adaptation standards specified in Handbook Version
1.0 (found in Section 4.1.6 of this version) and recommended increased
inclusivity, particularly with respect to cultural adaptations. The
Prevention Services Clearinghouse sought input from a range of experts
specifically focusing on program or service adaptations, including
those with expertise in cultural adaptations designed to serve
historically underserved communities. Underserved communities, as
articulated in the Executive Order on Advancing Racial Equity and
Support
[[Page 73024]]
for Underserved Communities Through the Federal Government, include
Black, Latino, and Indigenous and Native American persons, Asian
Americans and Pacific Islanders and other persons of color; members of
religious minorities; lesbian, gay, bisexual, transgender, and queer
(LGBTQ+) persons; persons with disabilities; persons who live in rural
areas; and persons otherwise adversely affected by persistent poverty
or inequality.
To meet the eligibility criteria of being clearly defined and
replicable, a program or service must have publicly available written
or recorded protocols, manuals, or the documentation (hereafter
referred to as ``manuals'') that describe how to implement the practice
(Section 2.1.2). A new section (2.3.2) clarifies the procedures used to
identify and review relevant manuals for a program or service. This
includes procedures for identifying a primary manual for review and
addressing cases with multiple potential manuals.
Many programs and services have multiple manuals, including manual
editions (e.g., editions of a manual as a program or service evolves
over time or expands) and manual variants (e.g., adaptations of a
program or service or a manual to address new issues, different
populations, or alternative approaches to delivering the program or
service). This section clarifies the standard process by which the
Prevention Services Clearinghouse assesses whether alternative manual
editions or variants have any substantial adaptations, compared to the
primary manual identified. This process consists of the following
steps, followed as needed based on the nature of the program or
service:
<bullet> Step 1: Determining whether the adaptation is explicitly
prohibited in the primary program or service manual under review or is
the result of adding another separate program or service to the
existing program or service (i.e., ``bundling'');
<bullet> Step 2: Determining whether the adaptation is explicitly
allowed by the primary program or service manual under review;
<bullet> Step 3: Determining whether the adaptation substantially
changes a program element in the primary program or service manual
under review;
<bullet> Step 4: Gathering additional information and consulting
with senior content experts on the Clearinghouse.
A revised table (Exhibit 2.4) classifies program elements and gives
examples of acceptable and substantial adaptations--including expanded
examples of adaptations that may be made in the process of culturally
adapting a program or service. (These criteria and procedures are
aligned with those used to assess any program or service adaptations
identified in studies during the study eligibility process, described
in Section 4.1.9). Manuals that are substantially adapted from a
primary manual may be considered as a separate program or service when
reviewing studies. Studies with these substantial adaptations would be
ineligible in a review based on the primary manual identified for a
particular program or service. Alternatively, manuals without
substantial adaptations may be considered the same program or service
when reviewing studies. Studies without substantial adaptations would
be included in a review based on the primary manual.
3.4 Chapter 3. Literature Search
To help ensure identification of studies conducted with American
Indian and Alaska Native populations, the draft Handbook Version 2.0
adds Healthy Native Youth to its list of clearinghouses used to
identify relevant research. The list of bibliographic databases has
been trimmed for efficiency and resource considerations. Some databases
in Handbook Version 1.0 were largely providing duplicative results.
This section clarifies that any publicly available research from
program or service websites is incorporated into the search.
Clarification is also provided on procedures for incorporation of
research that is submitted to the Prevention Services Clearinghouse
inbox ad hoc or during public calls.
3.5 Chapter 4. Study Eligibility Screening and Prioritization
3.5.1 Revision to Study Definition (Section 4.1)
In alignment with other Federal evidence clearinghouses, the
Prevention Services Clearinghouse intends to focus on degree of sample
overlap in applying its definition of a study as ``one research
investigation of a defined subject sample, and the interventions,
measures, and statistical analyses applied to that sample.'' Additional
study definition criteria (based on the What Works Clearinghouse v4.0
study definition) in Handbook Version 1.0 have been dropped in the
draft Handbook Version 2.0.
3.5.2 Clarifications on Source of Publication Criteria (Section 4.1.2),
Language of Publication (Section 4.1.3) and Location of Study (Section
4.1.4)
The draft Handbook Version 2.0 clarifies the definition of
``publicly available'' and ``published'' for the source of publication
standard (Section 4.1.2), in response to public comments.
Dissertations, theses, and conference papers remain ineligible. Given
the priority of reviewing a large number of programs and services, the
Prevention Services Clearinghouse intends to continue to exclude such
sources in the interests of efficiency.
Some public comments indicated confusion about whether studies
conducted outside of the United States or those conducted in non-
English-speaking countries are eligible. The draft Handbook Version 2.0
clarifies that the standard from Handbook Version 1.0 that studies must
be available in English (Section 4.1.3) is inclusive of studies
originally published in another language that have published English
language translations available. The draft Handbook Version 2.0
explicitly clarifies that studies conducted in any country are eligible
(Section 4.1.4), as they were under Handbook Version 1.0.
3.5.3 Revisions to Study Design and Intervention Condition Criteria
(Sections 4.1.5, 4.1.6)
The draft Handbook Version 2.0 provides clarification on
definitions for randomized group designs and quasi-experimental group
designs with respect to eligible study designs (Section 4.1.5). It
clarifies that single-group pretest-posttest designs and interrupted
time series designs without comparison groups are not eligible. It also
clarifies that group assignment must be exclusive for an outcome
measured at a given point in time--that is, participants cannot be
counted in both the intervention and comparison condition. The
criterion for eligible intervention conditions--that the intervention
group is offered an eligible program or service that is essentially the
same for all participants in the group--remains the same as in Handbook
Version 1.0, with minor clarifications, but is presented as a distinct
subsection in the draft Handbook Version 2.0 (Section 4.1.6) for
clarity.
3.5.4 Revisions to Eligible Comparison Conditions (Section 4.1.7)
Many public comments requested expansion of eligible study
comparison conditions beyond no or minimal treatment and treatment as
usual to
[[Page 73025]]
include more active comparison conditions. Many experts also
recommended that the Prevention Services Clearinghouse consider
including active comparison conditions. One consideration voiced by
multiple experts consulted is that active comparison conditions are
increasingly recommended, particularly if there are other available
interventions considered to be efficacious. Revision to this standard
was considered in the context of the FFSPA legislative criterion that a
program or service must be demonstrated as being superior to an
appropriate comparison practice.
The draft Handbook Version 2.0 allows for five types of eligible
comparison conditions:
<bullet> No intervention or wait list--offered no services or
services at a later date (clarifying that outcomes measured after a
wait list group is offered the intervention are not eligible).
<bullet> Minimal intervention--including informational materials or
psychoeducation, referrals to available services, or similar nominal
services.
<bullet> Placebo or attention control--conditions designed to
account for nonactive effects of treatment, such as participants'
expectations, contact time with an interventionist, or the relationship
between interventionist and participants; includes psychological or
pharmacological placebos, attention placebos, and nonspecific therapy
in which participants receive the same or similar amount of attention
or contact as the participants in the intervention condition.
<bullet> Treatment as usual--The draft Handbook Version 2.0
clarifies that both ``usual or typical services'' (i.e., individuals do
not receive anything they would not have been able to receive outside
the context of the study) or ``services consistent with usual or
typical services'' (i.e., services as part of the study that are not
offered in the community but are clearly described as consistent with
the usual or typical services that would be received by individuals or
families similar to those in the study) are considered eligible under
treatment as usual. Therapeutic or pharmacological interventions that
meet the definition of treatment as usual are eligible.
<bullet> Head-to-head comparisons--assigned to another intervention
that is not a variant of the program or service under review (may also
be referred to as alternative interventions, active interventions, or
comparator interventions); excluded are comparisons to pharmacological
interventions that do not meet the definition of treatment as usual
above.
The draft Handbook Version 2.0 indicates three types of comparison
conditions that are explicitly not eligible for review and provides a
rationale for each:
<bullet> Intervention variants--assigned to an intervention that is
a variation of the intervention under review. Examples include
dismantling studies (e.g., full version of intervention compared to one
lacking one or more components); bundled intervention studies (e.g.,
full version of intervention compared to a version with a second
intervention added); studies comparing different delivery modes,
providers, dosage, or fidelity levels for the same intervention;
sequencing studies (e.g., both conditions receive the same
interventions, but in a different order).
<bullet> Population-level data or benchmarks--constructed from
population norms or statistics derived from other studies, surveys,
censuses, or similar sources.
<bullet> Comprised only of intervention refusers or dropouts--
composed entirely of individuals who were offered the intervention
condition but refused the offer or dropped out of the intervention
after being offered the intervention.
3.5.5 Revisions to Outcomes (Section 4.1.8)
Definitions of outcome domain, outcome, and outcome measurement
have been provided for clarity. Clarifications have been included
regarding eligible outcomes within the child safety and child
permanency outcome domains and family functioning outcomes within the
adult well-being outcome domain. The clarifications to the child safety
and child permanency outcomes were previously described in the FAQ
section of the Prevention Services Clearinghouse website. Additionally,
eligible educational achievement and attainment outcomes in the child
well-being outcome domain have been expanded to include school
attendance and absenteeism as eligible outcomes. These outcomes, though
not direct measures of educational achievement and attainment, are
viewed as closely related and relevant outcomes. Clarification is
provided that outcomes that are composites of one or more eligible
outcomes within the eligible outcome domains are eligible; those that
are composites of eligible and ineligible outcomes are not eligible.
Clarification is also provided that eligible outcomes and outcome
measures may be defined differently across studies to reflect the
different ages, backgrounds, cultures, locations, and contexts of the
study participants, with examples provided.
The Prevention Services Clearinghouse currently does not have
measurement standards for assessing the validity or reliability of
biomarker measures (i.e., a physiological measure used as an indicator
of a physical, psychological or emotional state), such as the use of
cortisol as a measure of psychological stress. Expert consultations on
biomarkers did not indicate a clear set of standards that could be
broadly applied for review of such measures. As a result, the draft
Handbook Version 2.0 indicates that biomarker measures are not
currently eligible for review as child well-being or adult well-being
outcomes.
3.5.6 Revisions to Study Program or Service Adaptations Criteria
(Section 4.1.9)
Consistent with Handbook Version 1.0, the draft Handbook Version
2.0 indicates that, to be eligible for review, studies of a program or
service must all represent similar implementations of the program or
service selected for review. Revisions in the draft Handbook Version
2.0 clarify that the process of assessing program or service
adaptations for study eligibility is based on having identified a
particular manual (or set of manuals) of the program or service under
review (see Sections 2.3.1, 2.3.2).
The standard process used to identify whether program or service
adaptations are present in the studies being screened for eligibility
is clarified. The procedures and criteria for assessing whether
adaptations identified in studies are acceptable or substantial mirror
those specified in Section 2.3.2 for adaptations found in manual
editions or variants. The end result of these procedures is the
determination of study eligibility for the particular program or
service under review (in Section 2.3.2, the end determination is
whether two manuals are substantively similar or represent different
programs or services). Studies with any substantial adaptations are
ineligible for review as a study of the program or service under review
(such studies may be eligible for review as a study of different
program or service and its associated manual). Studies with only minor
adaptations may potentially be eligible if all other study eligibility
criteria are met.
3.5.7 Revisions to Study Review Prioritization Criteria (Section 4.2)
The Prevention Services Clearinghouse notes that study
prioritization criteria are distinct from study eligibility criteria.
When a
[[Page 73026]]
program or service has more than 15 studies eligible for review, study
prioritization criteria are applied to order the review of eligible
studies. The study prioritization process ensures efficiencies in the
reviewing process to review a large number of programs and services.
The Prevention Services Clearinghouse notes that only 12 of the 148
programs and services reviewed as of July 2023 had more than 15
eligible studies identified, requiring the use of study prioritization
criteria in these reviews to prioritize the first 15 eligible studies
for review using the design and execution standards. Of these 12
programs and services, nine had 16 to 25 eligible studies, with a few
having a much larger number of eligible studies (e.g., 75 or 90). All
other programs and services reviewed had 15 or fewer eligible studies,
with all eligible studies reviewed using the design and execution
standards. Therefore, as in Handbook Version 1.0, the study
prioritization criteria continue to apply only when there are 15 or
more eligible studies of a program or service in the draft Handbook
Version 2.0.
Three modifications have been made to the process of assigning
prioritization points for identifying the order in which studies are
reviewed in the draft Handbook Version 2.0. First, given that programs
or services must demonstrate sustained favorable effects 6 or 12 months
beyond the end of treatment (Section 7.2.3) to receive a rating of
supported or well-supported, the Prevention Services Clearinghouse
intends to increase the prioritization points given to studies that
include outcomes measured 6 or 12 months beyond the end of treatment to
ensure that these studies are reviewed earlier when present, increasing
the prioritization points for such studies to 3 and 6 points,
respectively (compared to 1 and 2 points, respectively, in Handbook
Version 1.0). Second, some public commenters and experts consulted
noted the importance of statistical power for being able to detect
intervention effects. The draft Handbook Version 2.0 adds one
prioritization score point for studies that report an analysis of
statistical power. Third, many public comments recommended that points
be awarded to studies based on populations served. The draft Handbook
Version 2.0 intends to add one prioritization score point for the child
welfare relevance of populations served and two prioritization points
for studies with samples from underserved communities. Prioritization
points for studies with outcomes in multiple outcome domains have been
decreased from a maximum of three to a maximum of one. The draft
Handbook Version 2.0 provides procedural details clarifying how ties in
prioritization scores are resolved in cases where more than 15 eligible
studies are identified.
The draft Handbook Version 2.0 includes efficiency enhancements
based on the study prioritization process for programs and services
where more than 15 eligible studies are identified. If, after review of
the first 15 eligible studies prioritized for review, a program or
service has not achieved a rating of well-supported, additional studies
are reviewed using the design and execution standards in their
prioritized order until either no eligible studies remain that could
result in further improvement to the program or service rating or all
eligible studies have been reviewed. Determination of potential for
program or service ratings to improve upon review of additional
eligible studies is based on (1) the program rating from studies
already reviewed using the design and execution standards and (2) the
duration of effects examined in the remaining studies (as assessed
according to study review prioritization criteria). Detailed examples
of the application of this policy are described in Section 4.2. The
draft Handbook Version 2.0 retains the policy from Handbook Version 1.0
of reviewing all studies against design and execution standards when 15
or fewer eligible studies are identified. All eligible studies are
reviewed for risk of harm.
3.6 Chapter 5. Evidence Review Using the Design and Execution Standards
3.6.1 Revisions and Clarifications to Contrasts Rated, Design and
Execution Rating Categories, Method of Assignment, and Integrity of
Random Assignment (Sections 5.1 to 5.5)
The draft Handbook Version 2.0 indicates that contrasts from all
eligible comparison conditions (Section 5.1) will be rated, whereas
under Handbook Version 1.0, only contrasts from the least-intensive
eligible comparison condition for a particular contrast were rated if
multiple comparison conditions were eligible for review (Handbook
Version 1.0, Section 4.1.4). Given the priority of reviewing a large
number of programs and services, the draft Handbook Version 2.0 retains
the policy from Handbook Version 1.0 of only reviewing full-sample
analyses and not reviewing subgroup or sensitivity analyses due to
resource considerations. For any studies that receive a moderate or
high design and execution rating and report subgroup analyses, the
Clearinghouse intends to indicate whether subgroup analyses were
conducted for informational purposes only. New and revised examples are
provided to clarify integrity of randomization standards for individual
and cluster-assignment designs.
3.6.2 Revisions and Clarifications to Attrition, Baseline Equivalence,
and Pretest Standards (Sections 5.6 to 5.8)
Based on expert feedback, and in alignment with other Federal
clearinghouses (in particular, the What Works Clearinghouse and Home
Visiting Evidence of Effectiveness [HomVEE]), the draft Handbook
Version 2.0 no longer requires baseline equivalence to be established
for a contrast from a low attrition randomized group design to receive
a ``High'' support of causal evidence rating.
Public comments expressed a desire for greater flexibility
regarding options for demonstrating baseline equivalence and
reconsideration of participant sociodemographic characteristics that
could be used to establish baseline equivalence when a pretest
alternative is not available. Informed by expert consultations, the
draft Handbook Version 2.0 maintains a general preference for using the
same (or nearly the same) measure as the outcome (i.e., a ``direct
pretest'') for baseline equivalence but now allows any eligible outcome
measure demonstrated to be correlated with the outcome at a threshold
of 0.60 or higher to be used to establish baseline equivalence (here
referred to as a ``correlated pretest measure''). Also informed by
expert feedback, when a correlated pretest measure or pretest
alternative is not available, the draft Handbook Version 2.0 provides
greater flexibility in the form of two options for establishing
equivalence on sociodemographic characteristics, allowing an expanded
set of individual characteristics and the use of a set of neighborhood
characteristics if only one individual characteristic is available.
Option 1 requires demonstration of baseline equivalence on at least two
of the following individual characteristics: race or ethnicity,
socioeconomic status, household composition, or age. If only one of the
four individual characteristics from Option 1 is available, baseline
equivalence can still be established under Option 2 if equivalence is
demonstrated on a measure of each of the following neighborhood
characteristics: race or ethnicity, socioeconomic status, and household
composition. When sociodemographics are used to establish
[[Page 73027]]
baseline equivalence, a new requirement indicates that study authors
must clearly describe all criteria used to create the intervention and
comparison groups and affirmatively indicate that the same or similar
criteria were used to create each group.
Binary measures have different statistical properties than
continuous measures that can potentially reduce their reliability as
indicators of baseline equivalence--particularly when events are rare
or in smaller samples. To address this, the draft Handbook Version 2.0
indicates a preference for continuous correlated pretests over direct
pretests when establishing baseline equivalence for a binary outcome.
It also permits use of continuous pretest alternative measures when
outcomes are binary, even if it was feasible to measure a direct
pretest. Specifically, continuous measures that meet the correlated
pretest measure or pretest alternative criteria are preferred over a
direct pretest of the binary measure, when available.
3.6.3 Revisions and Clarifications to Statistical Model Standards
(Section 5.9)
The statistical model standards (Section 5.9.1) have been revised
in the draft Handbook Version 2.0 to clarify procedures used when
statistical models do not meet standards and alternative statistical
models are not available or do not meet standards. In such cases, the
Prevention Services Clearinghouse will seek to review the contrast
based on unadjusted means and standard deviations and the statistical
significance test procedures specified in Chapter 6.
The measurement reliability standard for inter-rater reliability in
Handbook Version 1.0 was revised in the draft Handbook Version 2.0
(Section 5.9.2), with specific thresholds for inter-rater reliability
(correlation), inter-rater agreement on the basis of percentage
agreement (0.80 or higher), and inter-rater agreement based on kappa
(0.60 or higher). These revised standards are aligned with current What
Works Clearinghouse standards.
Some public comments expressed concern that confound standards
prevent inclusion of studies conducted in rural or underserved areas
where only a single service provider is available may not be able to
meet standards. The draft Handbook Version 2.0 clarifies that studies
with a single person or administrative unit are not automatically
confounded, with detailed clarifying examples added to this section.
Specifically, if a single provider (or a single administrative unit)
provides treatment or services to at least some participants in both
the intervention and comparison condition, a design confound is not
considered to be present. Expert feedback indicated that the confound
standards in Handbook Version 1.0 were appropriate causal evidence
standards, informing the retention of these confound standards in the
draft Handbook Version 2.0.
3.7 Chapter 6. Record and Characterize Impact Estimates
Public comments requested additional information about the formulae
used for computing effect sizes and procedures used for determining
statistical significance. The draft Handbook Version 2.0 provides all
standard formulae used in computing effect sizes reported and for
computing statistical significance. For models that meet statistical
model standards in the design and execution requirements (Section 5.9),
the draft Handbook Version 2.0 indicates that author-reported
statistical significance is preferred in covariate-adjusted models and
certain models for which the Prevention Services Clearinghouse does not
currently have standards for computing statistical significance (e.g.,
time-to-event models). When such models are not available or do not
meet statistical model standards, the formulae provided are used to
conduct a post-hoc statistical significance test based on the natural
metric of the outcome reported (e.g., continuous, binary, count, or
time-to-event).
Clarification is provided on information needed and procedures used
to compute effect sizes and statistical significance for repeated
measures models (e.g., growth curve models). In alignment with other
Federal clearinghouses (in particular, What Works Clearinghouse,
HomVEE), point-in-time estimates for each measurement time period are
required. If such information is not reported, unadjusted means and
standard deviations for each point in time are used (or requested if
not reported), with appropriate post-hoc significance tests performed
based on the natural metric of the outcome.
3.8 Chapter 7. Program or Service Ratings
3.8.1 Revisions and Clarifications to Program or Service Ratings
(Section 7.1) and Risk of Harm (Section 7.2.1)
No changes were made to the criteria for promising, supported, or
well-supported program or service ratings in the draft Handbook Version
2.0 (Section 7.1). This section clarifies that intention of the
Prevention Services Clearinghouse is for program or service ratings
from reviews conducted under Handbook Version 1.0 to be retained until
such time that a program or service is re-reviewed under Handbook
Version 2.0 (see Section 8.5.1 below regarding re-review procedures).
A new standard specified in the risk of harm section (Section
7.2.1) of the draft Handbook Version 2.0 is that contrasts in head-to-
head comparison conditions or placebo or attention control comparison
conditions where the comparison condition has any evidence for risk of
harm cannot contribute to a promising, supported, or well-supported
rating. If risk of harm is present in these kinds of comparison
conditions, impact estimates are not clearly interpretable as evidence
of intervention effectiveness--as it is possible that both the
intervention and comparison condition could be made worse off than if
they had not participated in the study at all. When risk of harm is not
present in the comparison condition, favorable impacts can be
interpreted as the intervention group being at least better off than
they would have been if no treatment had been offered at all and can
potentially contribute as evidence of effectiveness. Standard
procedures for identifying potential risk of harm in comparison
conditions are detailed in this section.
3.8.2 Revisions and Clarifications to Usual Care or Practice Settings
Definition (Section 7.2.2)
The definition of usual care or practice settings (Section 7.2.2)
in the draft Handbook Version 2.0 has been clarified to indicate that
community settings, such as schools, with embedded service providers
that may provide eligible programs or services as part of their typical
operations (e.g., school counselors), are also considered usual care or
practice settings. It clarifies that clinics that provide services
solely for participants in research studies or clinical trials (i.e.,
that do not provide any services to persons not participating in
research studies as part of their typical operations) do not constitute
usual care or practice settings.
3.8.3 Revisions and Clarifications to Beyond the End of Treatment
(Section 7.2.3)
Some public comments requested clarification on how the Prevention
Services Clearinghouse assesses the duration of sustained effects,
particularly in cases where the end of
[[Page 73028]]
treatment is flexible across participants. Section 7.2.3 of the draft
Handbook Version 2.0 includes revisions to clarify the order of
preference for information that may be provided in studies about the
end of treatment and procedures for computing the duration of sustained
effects when the duration of treatment is fixed, when the duration of
treatment is defined and varies across participants, and when the
duration of treatment is undefined. Treatment of boosters in computing
the duration of sustained effects is now explicitly addressed. Detailed
procedures and examples can be found in Section 7.2.3 of the draft
Handbook Version 2.0.
3.9 Chapter 8. Prevention Services Clearinghouse Procedures
The draft Handbook Version 2.0 represents the first update to the
Handbook of Standards and Procedures since the beginning of the Title
IV-E Prevention Services Clearinghouse in 2018. The basic procedures
for identifying eligible studies (Section 8.3) and reviewing studies
against the design and execution standards (Section 8.4) remain
essentially the same, with minor clarifications to operational
procedures. Author query policies (Section 8.4.2) have been clarified;
new content has been added clarifying the reasons that the Prevention
Services Clearinghouse may query program and service developers for
information about programs or services (Section 8.4.3). New content and
more substantive revisions are described below.
3.9.1 Selection of Handbook of Standards and Procedures Version To Use
in Reviews (Section 8.2)
The intention of the Prevention Services Clearinghouse is to
conduct reviews of any program or service not previously reviewed under
Handbook Version 1.0 solely under the standards and procedures
specified in Handbook Version 2.0 once it is finalized. Programs or
services that are included on the working list prior to when Handbook
Version 2.0 is finalized may be reviewed under Handbook Version 1.0 or
Handbook Version 2.0. The version of the handbook used to conduct a
review (or re-review) of a program or service will be clearly stated on
the working list and on the program or service's review page on the
Prevention Services Clearinghouse website.
3.9.2 Program and Service Re-Reviews and Study Re-Reviews (Sections
8.5.1, 8.5.2)
The Prevention Services Clearinghouse intends to conduct program
and service re-reviews solely under Handbook Version 2.0 after it is
finalized (Section 8.5.1). The intention of the Prevention Services
Clearinghouse is that all existing program and service ratings
determined under Handbook Version 1.0 will remain in effect until such
time that a program or service re-review is conducted of a program or
service.
Programs and services reviewed by the Prevention Services
Clearinghouse under Handbook Version 1.0 may be considered for re-
review under Handbook Version 2.0 if a re-review has the potential to
change the program or service rating (Section 8.5.1). Program or
service ratings could potentially change due to application of Handbook
Version 2.0 standards to studies already identified in a prior review
(e.g., studies previously ineligible now being eligible; studies being
able to demonstrate baseline equivalence under revised standards) or
the emergence of new evidence since the original review. The intention
of the Prevention Services Clearinghouse is that the rating of a re-
reviewed program or service would be based solely on the standards and
procedures in Handbook Version 2.0 (i.e., the previously assigned
rating would no longer be in effect).
The intention of the Prevention Service Clearinghouse is to conduct
study re-reviews (i.e., due to missing information or errors in the
currently published review of an individual study) under the version of
the handbook used to review the program or service (Section 8.5.2).
That is, for a program or service reviewed under Handbook Version 1.0
where the program or service has not been re-reviewed under Handbook
Version 2.0, a study re-review would be conducted under Handbook
Version 1.0. For a program or service where a program or service rating
has been assigned using Handbook Version 2.0, study re-reviews would be
conducted using Handbook Version 2.0. This policy is consistent with
other Federal evidence clearinghouses with multiple handbook versions
(e.g., HomVEE). The Prevention Services Clearinghouse's intention is
that the emergence of substantial new evidence that has the potential
to change program or service ratings (e.g., a newly published study)
should be addressed through a program or service re-review. Similarly,
cases where application of Handbook Version 2.0 standards to a study
reviewed under Handbook Version 1.0 could affect the program or service
rating are intended to be addressed through a program or service re-
review. Study re-reviews are intended to be limited solely to
addressing missing information or errors in studies already reviewed.
3.9.3 Manual Citation Updates (Section 8.5.3)
The Prevention Services Clearinghouse recognizes that program or
service manuals may be updated in the course of time after a review of
a program or service has been published. Should a new manual edition
(as defined in Section 2.3.2) be published, the public may request
consideration of an update to the manual citation used for the program
or service as outlined in Section 8.5.3 of the draft Handbook Version
2.0. If updated manual editions do not have substantive modifications
or adaptations from the manual reviewed (per the criteria specified in
Section 2.3), a manual citation may be updated to reflect that a newer
manual edition is in active use that is substantively similar to the
original primary manual selected for the review of the program or
service. In considering whether an update to a manual citation is
warranted, the Prevention Services Clearinghouse must have sufficient
information available to be able to apply the procedures specified in
Section 2.3 for determining whether any substantive adaptations are
present in the newer manual edition compared to the original edition
reviewed. If the manual citation is updated, the original manual
citation used to conduct the review of evidence for the program or
service will also be noted for clarity.
4.0 Timeline for the Clearinghouse To Apply New Standards and
Procedures
The Prevention Services Clearinghouse proposes to apply the
standards and procedures upon publication of a final Handbook Version
2.0. The public will be clearly notified on the Prevention Services
Clearinghouse website and via other avenues (e.g., email to subscribers
to the Prevention Services Clearinghouse email list) when the final
published Handbook Version 2.0 will go into effect for reviewing
programs and services.
Per the procedures in Chapters 7 and 8 of the draft Handbook
Version 2.0, all existing program and service ratings established under
Handbook Version 1.0 will remain in effect until such time that a
program or service re-review is conducted of a program or service under
Handbook Version 2.0.
5.0 Request for Information (RFI)
To facilitate the review of submissions, please identify the
chapter, section, and/or page number of the draft Handbook of Standards
and Procedures, Version 2.0 (https://
[[Page 73029]]
preventionservices.acf.hhs.gov/resources/comment-draft-handbook) that
your comments address. This RFI is for information and planning
purposes only and should not be construed as a solicitation or as an
obligation on the part of ACF or HHS. For more information about the
Prevention Services Clearinghouse, visit: <a href="https://preventionservices.acf.hhs.gov">https://preventionservices.acf.hhs.gov</a>.
Lauren Supplee,
Deputy Assistant Secretary for Planning, Research, and Evaluation.
[FR Doc. 2023-23391 Filed 10-23-23; 8:45 am]
BILLING CODE 4184-01-P
</pre><script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script></body>
</html>Indexed from Federal Register on October 24, 2023.
This is legal information, not legal advice. Laws vary by jurisdiction and change frequently. Always verify current law with official sources and consult a licensed attorney in your jurisdiction for advice on your specific situation.