Notice2021-17339
HUD Program Evaluation Policy-Policy Statement
Primary source
Metadata and text below are from the Federal Register, a public-domain U.S. government work. Always verify the official published version before relying on it for any legal matter.
Published
August 13, 2021
Issuing agencies
Housing and Urban Development Department
Abstract
This policy statement articulates the core principles and practices of the Department of Housing and Urban Development's evaluation and research activities. This policy reaffirms HUD's commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice.
Full Text
<html>
<head>
<title>Federal Register, Volume 86 Issue 154 (Friday, August 13, 2021)</title>
</head>
<body><pre>
[Federal Register Volume 86, Number 154 (Friday, August 13, 2021)]
[Notices]
[Pages 44738-44740]
From the Federal Register Online via the Government Publishing Office [<a href="http://www.gpo.gov">www.gpo.gov</a>]
[FR Doc No: 2021-17339]
=======================================================================
-----------------------------------------------------------------------
DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT
[Docket No. FR-6278-N-01]
HUD Program Evaluation Policy--Policy Statement
AGENCY: Office of Policy Development and Research, HUD.
ACTION: Notice.
-----------------------------------------------------------------------
SUMMARY: This policy statement articulates the core principles and
practices of the Department of Housing and Urban Development's
evaluation and research activities. This policy reaffirms HUD's
commitment to conducting rigorous, relevant evaluations and to using
evidence from evaluations to inform policy and practice.
DATES: August 13, 2021.
FOR FURTHER INFORMATION CONTACT: For information about this notice,
contact Todd M. Richardson, Evaluation Officer, Office of Policy
Development and Research, U.S. Department of Housing and Urban
Development, 451 7th Street SW, Washington, DC 20410, telephone (202)
402-5922. The listed telephone number is not a toll-free number.
Persons with hearing- or speech-impairments may access this number
through TTY by calling Federal Relay Service at 1-800-877-8339 (this is
a toll-free number).
SUPPLEMENTARY INFORMATION:
I. Background
Evaluation activity occurs in several offices at HUD, but the
special mission of HUD's Office of Policy Development and Research
(PD&R) is to inform HUD policy development and implementation to
improve life in American communities through conducting, supporting,
and sharing research, surveys, demonstrations, program evaluations, and
best practices. Within HUD, PD&R is responsible for most, but not all,
program evaluations. The office provides reliable and objective data
and analysis to help inform policy decisions.
In July 2016, GAO issued a report entitled ``Department of Housing
and Urban Development: Actions Needed to Incorporate Key Practices into
Management Functions and Program Oversight,'' (GAO 16-497) in which GAO
presented a broad assessment of HUD's management of its operations and
programs.\1\ In the report, GAO examined HUD efforts to: (1) Meet
Federal requirements and implement key practices for management
functions, including performance planning and reporting and human
capital, financial, acquisition, and information technology (IT)
management; and (2) oversee and evaluate programs.
---------------------------------------------------------------------------
\1\ See <a href="http://www.gao.gov/assets/680/678551.pdf">http://www.gao.gov/assets/680/678551.pdf</a>.
---------------------------------------------------------------------------
PD&R is the primary office within HUD responsible for data
analysis, research, program evaluations, and policy studies that inform
the development and implementation of programs and policies across HUD
offices. PD&R undertakes program evaluations, often by using a process
that includes convening expert panels. However, GAO found that PD&R had
neither developed agency-wide, written policies for its program
evaluations, nor documented the criteria used to select the expert
panels and review the quality of program evaluations.
On December 6, 2016 (81 FR 87949), HUD issued a policy statement in
the Federal Register \2\ responding to the GAO report by setting out
the core principles and practices of PD&R's evaluation and research
activities. This statement incorporated some language from a policy
statement by the Office of Policy, Research, and Evaluation of the
Administration for Children and Families of the U.S. Department of
Health and Human Services.
---------------------------------------------------------------------------
\2\ <a href="https://www.federalregister.gov/d/2016-29215">https://www.federalregister.gov/d/2016-29215</a>.
---------------------------------------------------------------------------
On January 14, 2019, the Foundations for Evidence-Based
Policymaking Act of 2018 (``Evidence Act''), Public Law 115-435, was
enacted.\3\ Section 101 of the Evidence Act created 5 U.S.C. 311-315
and mandated that the head of each agency appoint an Evaluation
Officer, including at HUD. This officer must establish common standards
for all HUD evaluations, whether performed by PD&R or another office.
This issuance articulates department-wide evaluation standards and
states other new principles based on PD&R's experience since the
November 2016 publication.
---------------------------------------------------------------------------
\3\ <a href="https://www.congress.gov/115/plaws/publ435/PLAW-115publ435.pdf">https://www.congress.gov/115/plaws/publ435/PLAW-115publ435.pdf</a>. See also the website, at <a href="https://www.congress.gov/bill/115th-congress/house-bill/4174">https://www.congress.gov/bill/115th-congress/house-bill/4174</a>, on the bill, H.R. 4174 of the
115th Congress, that became the Act.
---------------------------------------------------------------------------
II. HUD Program Evaluation Policy
Section 101 of the Evidence Act defines ``evaluation'' to mean ``an
assessment using systematic data collection and analysis of one or more
programs, policies, and organizations intended to assess their
effectiveness and efficiency.''
HUD has identified the following core principles and practices as
fundamental to ensuring high-quality and consistent evaluation results:
Rigor, relevance, transparency, independence, ethics, and technical
innovation. This policy applies to all HUD-sponsored
[[Page 44739]]
evaluations and regulatory impact analyses; they also apply to the
selection of projects, contractors, and HUD staff that are involved in
evaluations.
Rigor
HUD is committed to using the most rigorous methods that are
appropriate to the evaluation questions and feasible within budget and
other constraints. The need for rigor is not restricted to impact
evaluations; rigor is also necessary in implementation or process
evaluations, descriptive studies, outcome evaluations, and formative
evaluations; in both qualitative and quantitative approaches. Rigor
requires ensuring that inferences about cause and effect are well
founded (internal validity); requires clarity about the populations,
settings, or circumstances to which results can be generalized
(external validity); requires that researchers seek to understand and
correct for implicit bias in the formulation of research questions and
methods; and requires the use of measures that accurately capture the
intended information (measurement reliability and validity). Implicit
biases are discriminatory biases that reflect unidentified traces of
past experience, including implicit attitudes that affect feelings,
thoughts or actions, and implicit stereotypes that affect how others
are characterized. Survey instruments are pre-tested with members of
the population to be studied to increase measurement validity. When
statistically appropriate, and particularly if the number of hypotheses
being tested is large, HUD will require the use of commonly accepted
adjustments to classical statistical testing to reduce the probability
that random outliers are presented as meaningful.
In assessing the effects of programs or services, HUD evaluations
use methods that isolate to the greatest extent possible the impacts of
the programs or services from other influences, such as trends over
time, geographic variation, or pre-existing differences between
participants and non-participants. Where feasible, research should
employ a treatment group and a counterfactual. A treatment group is a
population that has received an intervention. A counterfactual is a
population that did not receive the intervention and can credibly
represent what would have happened to the treatment group in the
absence of the intervention according to the above standard for rigor.
For such causal questions, experimental approaches are preferred. When
experimental approaches are not feasible, HUD uses the most rigorous
approach that is feasible.
In both quantitative and qualitative research, rigor means having
clear research questions and an explicit analytic framework;
justification for case selection and sampling methods in relation to
research goals; and transparent, verifiable methods of systematic data
collection and analysis, auditable records, and attention to possible
alternative interpretations during analysis and writing.
HUD ensures that contractors and grantees conducting evaluations
have appropriate expertise through emphasizing the requirement for
rigor in requests for proposals and funding opportunity announcements,
noting that applicants' capacity for rigor will be evaluated in the
selection process. In addition, HUD will judge research teams with
equal capacity for rigor to be more qualified if the team includes
researchers demographically similar to or knowledgeable about the
perspectives and lived experiences of the populations studied.
HUD employs a strategic human capital development plan to hire,
train, and retain a workforce that ensures staff have the tools and
resources to accomplish the mission.
Relevance
The HUD evaluation agenda reflects the legislative requirements and
policy issues related to HUD's mission. HUD solicits input from
stakeholders, both internal and external, including stakeholders with
lived experience, such as program participants, and grantees, on the
selection of programs to be evaluated, initiatives, demonstrations, and
research questions. For new initiatives and demonstrations in
particular, evaluations will be more feasible and useful when planned
in advance, in concert with the development of the initiative or
demonstration, rather than as an afterthought. HUD strives to
understand the relevance of its completed research through concerted
stakeholder engagement, including with people and grantees affected by
HUD programs, to continuously improve its research agenda.
Expert panels include research and other subject matter experts and
are diverse in ways tailored to the study, including racial and ethnic
diversity and representatives of the studied populations.
HUD strives to design program evaluations and other analyses to
better understand structural racism and to reveal unequal benefits and
harms across social groups as relevant, with special attention to race,
national origin, color, familial status, religion, disability, age, and
sex (including gender identity and sexual orientation).
To support this goal, insofar as feasible, HUD collects and reports
data on race, ethnicity, gender, and income, and other characteristics
of underrepresented and underserved communities relevant for research
and analysis efforts.
HUD also encourages research to engage studied populations.
``Engagement'' means the deliberate and intentional inclusion of the
thoughts and perspectives of studied groups, such as program
participants, grantees, and underrepresented and underserved
populations. This includes collecting people's thoughts and
perspectives through standard (valid and rigorous) research methods
such as surveys, focus groups, in-depth interviews, or ethnography to
produce published research that conveys studied populations' thoughts
and perspectives. In another type of engagement, people influence the
research that is about them. This includes a wide range of activities
that lie on a continuum from simple input (which should be documented
and published) to full co-creation of any aspect of the research, from
topic selection to research design, data collection, data analysis,
interpretation, writing, or dissemination, or even being on the
research team. An example of simple input is to include in the final
report a summary of comments on the research by members of studied
groups. Examples of collaboration include, but are not limited to, co-
creating the list of topics to be covered in a survey or having members
of a studied group on the research team.
To raise awareness of and spur creative approaches to engagement of
studied populations in program evaluations and HUD-sponsored research,
HUD may require contractors and grantees to explain how their research
will and will not engage studied populations. HUD recognizes that
engagement must be tailored to particular research efforts.
HUD retains the right to determine research methods.
HUD disseminates findings in ways that are accessible and useful to
policymakers, practitioners, and members of communities affected by HUD
programs and policies. Published findings will be accessible to
individuals with disabilities pursuant to Section 508 of the
Rehabilitation Act. PD&R partners with other HUD program offices to
inform internal and external stakeholders through disseminating
[[Page 44740]]
evidence from HUD-sponsored evaluations.
Transparency
HUD will release methodologically valid evaluations without regard
to the findings. Evaluation reports must describe the methods used,
including strengths and weaknesses, and discuss the generalizability of
the findings. Evaluation reports must present comprehensive results,
including favorable, unfavorable, and null findings. HUD will publish
interim findings, as projected in the initial research design. If there
are indications that the findings of the final report may differ, HUD
will provide appropriate qualifications accompanying the publication to
guard against misunderstanding or misuse of the interim findings. If
there are interim findings, HUD will publish those findings even if
there are indications that the findings of the final report may differ.
When findings are highly relevant to current policy, HUD evaluations
carry a foreword articulating the policy position of the Department
with respect to those findings.
If the findings of a HUD evaluation will have broad public interest
and includes a counterfactual, PD&R will publish a synopsis of the
research design, data collection and analysis plan soon after it is
approved and will require interim and final reports that deviate from
that document to explain how they deviate and why.
HUD publishes a 5-year Learning Agenda \4\ that outlines the
research and evaluation that it believes would be of greatest value to
public policy. PD&R lists all ongoing evaluation projects at the
<a href="http://HUDUSER.gov">HUDUSER.gov</a> website \5\ and updates it quarterly. PD&R will release
evaluation results timely, usually within 4 months of receiving the
final report.
---------------------------------------------------------------------------
\4\ <a href="https://www.huduser.gov/portal/about/pdr_learningagenda.html">https://www.huduser.gov/portal/about/pdr_learningagenda.html</a>.
\5\ <a href="https://www.huduser.gov/portal/about/PDR-Research.html">https://www.huduser.gov/portal/about/PDR-Research.html</a>.
---------------------------------------------------------------------------
HUD will, where possible, archive administrative and evaluation
data for secondary use by interested researchers. HUD typically builds
requirements into contracts to prepare data sets for secondary use.
Access for external researchers may be provided directly through data
licenses or indirectly through inter-agency agreements. This policy may
not apply for data that has obvious commercial value, such as mortgage
performance data. HUD staff may publish the results of their
scholarship and analysis in any forum, so long as they do not claim to
speak for the Department.
HUD evaluation contracts will generally permit contractors to
publish their findings within 6 months of the termination of the
contract if HUD has not already published them.
Independence
Independence and objectivity are core principles of evaluation.
Agency and program leadership, program staff, service providers, and
others participate actively in setting evaluation priorities,
identifying evaluation questions, and assessing the implications of
findings. However, it is important to insulate evaluation functions
from undue influence and from both the appearance and the reality of
bias. To promote objectivity, HUD protects independence in the design,
conduct, and analysis of evaluations. To this end:
<bullet> HUD conducts evaluations through the competitive award of
grants and contracts to external experts who are free from conflicts of
interest.
<bullet> HUD also conducts evaluations in-house and supports
unsolicited external evaluation proposals with funding, data, or both.
<bullet> The Evaluation Officer will consult with the HUD office
with lead responsibility on the design of evaluation projects and
analysis plans and will advise that office on whether to publish
evaluation reports.
Ethics
HUD-sponsored evaluations must be conducted in an ethical manner
and safeguard the dignity, rights, safety, and privacy of participants.
HUD-sponsored evaluations must comply with both the spirit and the
letter of relevant requirements such as regulations governing research
involving human subjects. In particular, PD&R protects the privacy of
HUD-assisted households and HUD-insured borrowers through its Rule of
Eleven; that is, PD&R allows no disclosure of information about the
characteristics of any group of individuals or households numbering
fewer than eleven by PD&R staff, contractors, grantees, or licensees.
HUD is a signatory to the Federal Policy for the Protection of
Human Subjects, generally known as the ``Common Rule.'' 24 CFR part 60,
which includes its own requirements for ensuring adequate provisions to
protect the privacy of human subjects research.\6\
---------------------------------------------------------------------------
\6\ <a href="https://www.ecfr.gov/cgi-bin/retrieveECFR?n=pt24.1.60">https://www.ecfr.gov/cgi-bin/retrieveECFR?n=pt24.1.60</a>.
---------------------------------------------------------------------------
HUD does not tolerate plagiarism, or fabrication or deliberate
mischaracterization of data by staff, contractors or grantees who are
engaged in evaluation activity.
Technical Innovation
PD&R supports and employs new methods of data collection and
analysis that more reliably and efficiently answer research questions
than old methods do.
Application of These Principles to Economic Analysis of Regulations
Economic analysis of both existing and proposed regulations,
properly conducted, is a critical tool in improving public policy.
Economists at HUD rely on the insights, data, and empirical estimates
from rigorous program evaluations when predicting the economic impact
of an incremental change to the program. In any HUD Regulatory Impact
Analysis:
<bullet> HUD analyzes whether the issues addressed by the
regulation stem from a market failure, government failure, or other
systemic problem, and whether the regulation addresses the root causes
of those problems.
<bullet> HUD uses and as necessary produces the best objective
estimates of the benefits, costs, and transfers resulting from the
regulation, taking into account gaps and uncertainties in the available
data and methodologies.
<bullet> HUD assesses the economic benefits, costs, and transfers
of proposed regulatory actions as required by Executive Order 12866.\7\
HUD provides additional analysis of impacts across groups defined by
race, ethnicity, and other characteristics that may define
underrepresented and underserved groups when such analyses are relevant
and feasible.
---------------------------------------------------------------------------
\7\ See <a href="https://www.archives.gov/files/federal-register/executive-orders/pdf/12866.pdf">https://www.archives.gov/files/federal-register/executive-orders/pdf/12866.pdf</a> for the original order, and see
<a href="https://www.archives.gov/federal-register/executive-orders/1993-clinton.html">https://www.archives.gov/federal-register/executive-orders/1993-clinton.html</a> for citations of, and links to, other executive orders
that amended or supplemented this order.
---------------------------------------------------------------------------
<bullet> Where clear alternatives to the regulatory actions exist,
HUD objectively estimates the benefits, costs, and transfers of those
alternatives, and additional analysis of impacts of those alternatives
across underrepresented or underserved groups as well.
Todd M. Richardson,
Evaluation Officer.
[FR Doc. 2021-17339 Filed 8-12-21; 8:45 am]
BILLING CODE 4210-67-P
</pre></body>
</html>Indexed from Federal Register on August 13, 2021.
This is legal information, not legal advice. Laws vary by jurisdiction and change frequently. Always verify current law with official sources and consult a licensed attorney in your jurisdiction for advice on your specific situation.