Notice2025-08842
Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0
Primary source
Metadata and text below are from the Federal Register, a public-domain U.S. government work. Always verify the official published version before relying on it for any legal matter.
Published
May 19, 2025
Issuing agencies
Health and Human Services DepartmentChildren and Families Administration
Abstract
The Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), oversees the Pathways to Work Evidence Clearinghouse (Pathways to Work). ACF seeks comments on proposed revisions to existing standards and on new standards for including and rating the quality of program cost studies in the "Protocol for the Pathways to Work Evidence Clearinghouse: Methods and Standards, Version 2.0".
Full Text
<html>
<head>
<title>Federal Register, Volume 90 Issue 95 (Monday, May 19, 2025)</title>
</head>
<body><pre>
[Federal Register Volume 90, Number 95 (Monday, May 19, 2025)]
[Notices]
[Pages 21310-21311]
From the Federal Register Online via the Government Publishing Office [<a href="http://www.gpo.gov">www.gpo.gov</a>]
[FR Doc No: 2025-08842]
[[Page 21310]]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Protocol for the Pathways to Work Evidence Clearinghouse: Methods
and Standards, Version 2.0
AGENCY: Administration for Children and Families, Department of Health
and Human Services.
ACTION: Request for public comments.
-----------------------------------------------------------------------
SUMMARY: The Administration for Children and Families (ACF), within the
U.S. Department of Health and Human Services (HHS), oversees the
Pathways to Work Evidence Clearinghouse (Pathways to Work). ACF seeks
comments on proposed revisions to existing standards and on new
standards for including and rating the quality of program cost studies
in the ``Protocol for the Pathways to Work Evidence Clearinghouse:
Methods and Standards, Version 2.0''.
DATES: The deadline for comments on this notice is June 18, 2025.
ADDRESSES: Interested parties may submit written questions, comments,
and supplementary documents by email to
<a href="/cdn-cgi/l/email-protection#1e4e7f6a76697f676d5d727b7f6c77707976716b6d7b5e7f7c6a7972717c7f72307d7173"><span class="__cf_email__" data-cfemail="6c3c0d18041b0d151f2f00090d1e05020b0403191f092c0d0e180b00030e0d00420f0301">[email protected]</span></a> with ``Pathways to Work Evidence
Clearinghouse 2025 FRN comment'' in the subject line. To ensure that
your comments have maximum effect, please identify clearly the section
of the draft Protocol for the Pathways to Work Evidence Clearinghouse:
Methods and Standards, Version 2.0 that your comments address.
Readers are referred to the full version of the draft Protocol for
the Pathways to Work Evidence Clearinghouse: Methods and Standards,
Version 2.0 (Protocol Version 2.0) on the Clearinghouse website
(<a href="https://pathwaystowork.acf.gov/sites/default/files/2025-05/Pathways_to_Work_Methods_and_Standards_Report_V2.0.pdf">https://pathwaystowork.acf.gov/sites/default/files/2025-05/Pathways_to_Work_Methods_and_Standards_Report_V2.0.pdf</a>).
SUPPLEMENTARY INFORMATION:
1.0 Background and Legislative Context
In response to the Consolidated Appropriations Act of 2017 (Pub. L.
115-31), the ACF Office of Planning, Research, and Evaluation (OPRE)
established the Pathways to Work Evidence Clearinghouse (Pathways to
Work). Pathways to Work seeks to be a comprehensive resource that a
range of audiences, including State and local Temporary Assistance for
Needy Families administrators, can use to identify the services that
will best help people with low incomes succeed in the labor market.
Pathways to Work carries out a systematic process to identify, review,
and rate the quality of eligible studies of programs and policies that
have the primary aim of improving the employment and earnings of people
with low incomes. The process is implemented by trained reviewers using
consistent and transparent standards and procedures. The current
``Protocol for the Pathways to Work Evidence Clearinghouse: Methods and
Standards Report (Protocol Version 1.0)'' (<a href="https://pathwaystowork.acf.gov/publication/ProtocolPathways">https://pathwaystowork.acf.gov/publication/ProtocolPathways</a>) provides a
detailed description of the standards currently used to identify,
review, and rate the quality of eligible studies and the procedures
followed by Pathways to Work staff. The Protocol Version 1.0 was
informed by the scope and protocols used in a prior OPRE-sponsored
evidence review project, the Employment Strategies for Low-Income
Adults Evidence Review, consultations with research and practice
experts, and the review processes developed and used by other prominent
federally funded evidence clearinghouses. The Protocol Version 1.0 was
further informed by public comments submitted in response to Federal
Register Notice 83 FR 26290 (June 6, 2018), here: <a href="https://www.federalregister.gov/d/2018-12160">https://www.federalregister.gov/d/2018-12160</a>.
2.0 Revisions and Clarifications to the Pathways to Work Methods and
Standards in the Draft Protocol Version 2.0
The draft Protocol Version 2.0 aims to revise existing standards
and to add new standards for including and rating the quality of
program cost studies. The draft Protocol Version 2.0 was informed by
consultations with research and practice experts. Below is a
comprehensive list of specific revisions proposed to Pathways to Work's
existing methods and standards, and the new standards for reviewing
program cost studies. Subsequent chapter and section numbers all refer
to the chapter and section numbering for the draft Protocol Version
2.0, unless the text explicitly indicates a reference to Protocol
Version 1.0 chapter and section numbering.
2.1 Chapter 1. Introduction
The revised introduction includes an update of the key terms used
by Pathways to Work, reflecting engagement activities with key Pathways
to Work audiences to enhance plain language terminology. The content of
the section on updates to the report was moved to a revised Appendix A
describing the evolution of the protocol and methods and standards.
2.2 Chapter 2. Identifying Eligible Studies
Revisions to the electronic citation database section (Section
2.1.3) include dropping the set of intent and geographic terms and
enhancing the set of outcomes, sample, and design keywords to ensure
greater comprehensiveness, informed by testing the sensitivity of
results to these terms and expert consultations. The ProQuest
Dissertations and Theses database was removed from the protocol based
on very few eligible studies being identified through this database,
though dissertations and theses remain eligible for review. The
protocol for searching organizational websites is now in its own
section (Section 2.1.4), and the list of organizational websites
searched was updated (see Box 2.1). The Prevention Services
Clearinghouse was added to the list of federal evidence review
databases consulted (Section 2.1.5). The Protocol Version 2.0 indicates
an intent to move the list of literature searches identified from an
appendix in the protocol to the Pathways to Work website.
2.3 Chapter 3. Assessing a Study's Strength of Evidence
Two key revisions were made to the standards for reviewing
comparison-group, quasi-experimental designs (Section 3.2.5). First, in
determining whether the study controlled for any potential differences
in the outcome before the program, current standards require that the
control for employment or earnings outcomes should be measured at least
1 year before the intervention. The revised standards allow for an
exemption on this timing requirement in two specific cases: (1) studies
where participants have been incarcerated for 1 year or longer prior to
the program beginning; and (2) studies of minor youth who would not
have had sufficient time to establish 1 year or more of work history
(e.g., a program designed for 16-year-olds with no prior work history).
The timing requirement for the control to be measured at least 1 year
before the program began may also potentially be waived in cases where
authors provide explicit, credible evidence that participants in both
the intervention and comparison conditions in the study could not have
had a change in earnings in the year prior to the program beginning.
Waiving the timing requirement requires the approval of the Pathways to
Work principal investigator. Note that in all cases, this exemption
applies only to the
[[Page 21311]]
timing requirement for when potential differences were measured--not to
the requirement to control for any potential differences in the outcome
before the program began.
Second, baseline equivalence standards were revised to permit use
of effect sizes to establish baseline equivalence if there is a
statistically significant difference on one of the required
characteristics. If the baseline difference is statistically
significant, reviewers then compute the effect size of the baseline
difference, following the procedures specified in Appendix D of
Protocol Version 2.0. If the effect size is less than 0.05 standard
deviations, the baseline equivalence requirement is met. If the effect
size is between 0.05 and 0.25 standard deviations, the study must
control for the characteristic in their statistical model. If the
effect size is greater than 0.25 standard deviations (or if an effect
size cannot be computed), then the baseline equivalence requirement is
not met. This revision is intended to address studies with large sample
sizes where even very small differences in the magnitude of the
baseline difference may be statistically significant.
2.4 Chapter 4. Assessing the Evidence of Effectiveness for a Program
Revisions were made to the program effectiveness ratings, where the
criteria for the Insufficient Evidence to Assess Support rating were
revised to include only cases with a single study where none of the
findings are statistically significant. The criteria for Not Supported
did not change but now include any cases with two or more studies that
have a pattern of only null or unfavorable findings that do not meet
the criteria for any other rating category. Lastly, the name of the
rating category for cases with no high or moderate-quality studies
identified that was formerly titled No Evidence to Assess Support has
been changed to Cannot Assess Support for clarity.
2.6 Chapter 5. Assessing Cost Study Information for a Program
As the evidence base on employment programs for job seekers with
low incomes continues to grow, so has the need for information about
the costs of those programs and practices. Without additional
information about the personnel/non-personnel resources used and the
associated costs, it is impossible to provide guidance on the resources
necessary for implementation and how best to allocate funding towards
these efforts. Pathways to Work developed new standards for reviewing
cost studies on employment and training programs designed for
individuals with low incomes, with input from research and practice
experts.
The draft standards cover three types of cost studies. Cost
analysis provides an analysis of the comprehensive effort involved in
program implementation, answering questions such as how much the
program costs to implement, the per-participant cost of the program,
and the feasibility of implementation given existing budget constraints
and available resource inputs. Cost-effectiveness analysis compares the
estimated cost of a program with an estimate of its impact on a given
outcome of interest. Cost-benefit analysis compares the cost of a
program with the monetized outcomes associated with that program.
The draft cost standards include two types of standards: (1)
threshold standards that identify basic characteristics that cost
studies must meet in order to rate the quality of the study's cost
information, and (2) quality rating standards that are applied to
studies that meet the threshold standards and provide additional
information about how the study calculated program costs. Chapter 5
provides the threshold and quality standards for cost analyses. Studies
that include a cost-effectiveness analysis and/or cost-benefit analysis
will be reviewed and scored separately using the standards for each
type of analysis, respectively, presented in Appendix E. A cost study
rating will be assigned based on the threshold and quality rating
standards. Cost studies that do not meet all the threshold standards
receive a rating of Interpret cost findings with caution. Cost studies
that meet all threshold standards are scored on quality rating
standards on a 1 to 3 scale. A summative quality score is then
generated by averaging the quality rating scores from each individual
quality standard, which also ranges from 1 to 3, and rounding to the
nearest tenth. A rating of Cost study meets standards with low quality
will be assigned when the average score is between 1.0 and 1.5. A
rating of Cost study meets standards with moderate quality will be
assigned when the average score is between 1.6 and 2.5. A rating of
Cost study meets standards with distinction will be assigned when the
average score is between 2.6 and 3.0.
Detailed information about the proposed threshold standards for
cost analyses can be found in Section 5.1 of the draft Protocol Version
2.0. Detailed information about the proposed cost analysis quality
rating standards can be found in Section 5.2 of the draft Protocol
Version 2.0. Detailed information about cost-effectiveness analysis and
cost-benefit analysis threshold and quality standards can be found in
Appendix E.
3.0 Timeline for the Pathways to Work To Apply New Methods and
Standards
Pathways to Work proposes to apply the standards and procedures
upon publication of a final Protocol Version 2.0. The public will be
clearly notified on the Pathways to Work website and via appropriate
dissemination channels when the final published Protocol Version 2.0
will go into effect.
4.0 Request for Information (RFI)
ACF invites comments regarding this notice on the draft ``Protocol
for the Pathways to Work Evidence Clearinghouse: Methods and Standards,
Version 2.0'' (<a href="https://pathwaystowork.acf.gov/sites/default/files/2025-05/Pathways_to_Work_Methods_and_Standards_Report_V2.0.pdf">https://pathwaystowork.acf.gov/sites/default/files/2025-05/Pathways_to_Work_Methods_and_Standards_Report_V2.0.pdf</a>). To
facilitate the review of comments submitted, please identify the
chapter, section, and/or page number of the draft that your comments
address. This RFI is for information and planning purposes only and
should not be construed as a solicitation or as an obligation on the
part of ACF or HHS. For more information about the Pathways to Work,
visit: <a href="https://pathwaystowork.acf.gov/">https://pathwaystowork.acf.gov/</a>.
Lauren Supplee,
Deputy Assistant Secretary for Planning, Research, and Evaluation.
[FR Doc. 2025-08842 Filed 5-16-25; 8:45 am]
BILLING CODE 4184-09-P
</pre><script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script></body>
</html>Indexed from Federal Register on May 19, 2025.
This is legal information, not legal advice. Laws vary by jurisdiction and change frequently. Always verify current law with official sources and consult a licensed attorney in your jurisdiction for advice on your specific situation.