Revised Procedures and Standards: Home Visiting Evidence of Effectiveness (HomVEE) Review
| Published date | 05 August 2020 |
| Citation | 85 FR 47376 |
| Record Number | 2020-17001 |
| Section | Notices |
| Court | Children And Families Administration |
Federal Register, Volume 85 Issue 151 (Wednesday, August 5, 2020)
[Federal Register Volume 85, Number 151 (Wednesday, August 5, 2020)]
[Notices]
[Pages 47376-47384]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2020-17001]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Revised Procedures and Standards: Home Visiting Evidence of
Effectiveness (HomVEE) Review
AGENCY: Administration for Children and Families, U.S. Department of
Health and Human Services.
ACTION: Request for public comment.
-----------------------------------------------------------------------
SUMMARY: The Administration for Children and Families (ACF), within the
U.S. Department of Health and Human Services (HHS), oversees the Home
Visiting Evidence of Effectiveness
[[Page 47377]]
(HomVEE) review, which is proposing to revise the procedures and
standards that guide its work. The revised procedures and standards
will be presented in two separate Federal Register notices. The current
Federal Register notice seeks comments on proposed changes and
clarifications to several procedural topics and on the standards for
assessing the quality of impact study designs. Readers are referred to
the full version of the HomVEE Draft Version 2 Handbook on the HomVEE
website (https://homvee.acf.hhs.gov) for more details. Another Federal
Register notice summarizes updated definitions, rules, and procedures
related to handling home visiting model versions (commonly referred to
in the home visiting research literature as adaptations) in the review.
DATES: Send comments on or before September 1, 2020.
ADDRESSES: Submit questions, comments, and supplementary documents to
[email protected] with ``HomVEE procedures and standards FRN comment''
in the subject line.
SUPPLEMENTARY INFORMATION:
Invitation to Comment: HHS invites comments regarding this notice.
To ensure that your comments are clearly stated, please identify the
section of this notice or the chapter and section of the HomVEE Draft
Version 2 Handbook that your comments address.
1.0 Background
To help policymakers, program administrators, model developers,
researchers, and the public identify rigorous research and understand
which early childhood home visiting models are effective, ACF's Office
of Planning, Research, and Evaluation within HHS oversees the HomVEE
review. HomVEE's mission is to conduct a thorough and transparent
review of the research literature on home visiting for families with
pregnant women and children from birth to kindergarten entry. The
review team identifies well-designed research within that pool and
extracts and summarizes the findings from that research.
One critical use of HomVEE's results is to determine which home
visiting models meet the HHS criteria for an ``evidence-based early
childhood home visiting service delivery model'' (see Exhibit II.11 in
the HomVEE Draft Version 2 Handbook), a key requirement of eligibility
for implementation with the Maternal, Infant, and Early Childhood Home
Visiting (MIECHV) Program funding. The MIECHV Program is administered
by the Health Resources and Services Administration in partnership with
ACF. Created in 2010, the MIECHV Program provides funding to states,
territories, and tribal entities to implement home visiting models.
MIECHV awardees have the flexibility to tailor the program to serve the
specific needs of their communities. Through a needs assessment,
awardees identify at-risk communities and select home visiting service
delivery models that best meet state and/or local needs. As per
MIECHV's authorizing statute, state and territory awardees must spend
the majority of their MIECHV Program grants to implement evidence-based
home visiting models, with up to 25 percent of funding available to
implement promising approaches that will undergo rigorous evaluation.
For the first time since its inception in 2009, HomVEE is proposing
to revise the procedures and standards that guide the systematic
review. The proposed revisions include (1) clarifying and updating
standards and procedures for rating the quality of impact studies that
are used to determine which home visiting models meet HHS criteria for
an ``evidence-based early childhood home visiting service delivery
model'' and (2) clarifying definitions, rules, and procedures for
handling model versions (commonly referred to in the home visiting
research literature as adaptations) in the review (presented in a
separate Federal Register notice). The current Federal Register Notice
focuses on the former set of revisions.
The proposed changes to HomVEE procedures and standards generally
bring the review into alignment with procedures and standards for other
federally sponsored systematic evidence reviews. The proposed revisions
also specify standards for research designs that are becoming
increasingly common in home visiting studies. Over the course of the
past 2 years, HomVEE consulted with methodological experts and other
federal evidence reviews to refine and update the procedures and
standards.
Through this Federal Register notice, HomVEE seeks to provide a
transparent account of how the review operates and to gather
stakeholder input on draft changes. The sections below summarize the
main changes to the procedures and standards. A preliminary version of
these revisions, the HomVEE Draft Version 2 Handbook, is available
during the public comment period on the HomVEE website at (https://homvee.acf.hhs.gov).
After a period of public comment (including close consultation with
selected methods experts outside of HomVEE), HomVEE will release a
final Version 2 Handbook.
2.0 Changes and Clarifications to HomVEE Procedures
It is natural in the course of a systematic review for issues to
arise that cannot be addressed by existing procedures. These issues
require the HomVEE team to develop internal guidance to guide the
review.
Clarifications to the procedures added to the HomVEE Draft Version
2 Handbook, summarized in Sections 2.1 through 2.7 below, represent
HomVEE's attempt to formalize internal guidance generated over the
course of conducting reviews so that the procedures are applied
systematically.
Similarly, as reviews evolve, it is necessary for ongoing
systematic reviews to change their procedures to meet the needs of the
field. Changes are new procedures proposed to align with best practices
in systematic reviews and keep the HomVEE review current.
2.1 Clarify Definitions of Research Terms
Recognizing the importance of clear communication and consistent
terminology when applying systematic review rules, HomVEE clarifies the
definitions of important research terms in the HomVEE Draft Version 2
Handbook. Exhibit I.3 of the HomVEE Draft Version 2 Handbook presents
these, as listed below.
A study evaluates a distinct implementation of an
intervention (that is, with a distinct sample, enrolled into the
research investigation at a defined time and place, by a specific
researcher or research team). HomVEE reviews eligible manuscripts about
studies that examine the impact of an early childhood home visiting
model by comparing an intervention condition (in which study
participants are offered the home visiting model under study) and a
comparison condition (in which study participants are not offered that
model). This includes eligible manuscripts about studies on model
replications, iterations, and versions. See Chapter III, Section A.1.b
of the HomVEE Draft Version 2 Handbook, including Exhibit II.4, for
more information on how HomVEE screens research for eligibility.
[cir] A sample encompasses both the entire intervention group and
the entire comparison group of participants included in a study.
[cir] A subgroup is a subset of the sample examined in a study
(that is, an analytic subgroup). For example, researchers may examine
how a home visiting model affects teenage mothers when there are
mothers with a range of
[[Page 47378]]
ages in their study; hence, teenage mothers would be an analytic
subgroup. Sometimes, researchers present subgroup findings in a
manuscript alongside findings for the overall sample, and sometimes
researchers prepare a manuscript based exclusively on subgroup findings
from a broader study. (For HomVEE, results from teenage mothers would
not be considered an analytic subgroup analysis when the overall study
only enrolled teenage mothers.) See 2.7 below for more details on
HomVEE's clarified subgroup definition.
Manuscripts describe study results. Manuscripts may be
published or unpublished research, such as journal articles, book
chapters, or working papers. A single study may produce one, or many,
manuscripts. Typically, one manuscript reports on only one study,
although in rare cases one manuscript may include several studies, if
it describes evaluations of multiple interventions or the same
intervention evaluated in multiple distinct (non-overlapping) samples.
Findings summarize the effect of a home visiting model on
a specific sample or subgroup, on a specific eligible outcome measure
(see Chapter III, Section A.4.a of the HomVEE Draft Version 2
Handbook), at a specific time point, from a specific analysis. A
manuscript typically includes multiple findings.
HomVEE rates findings (according to standards proposed in Chapter
III of the HomVEE Draft Version 2 Handbook) and sorts manuscripts
according to the highest-rated finding in the manuscript (see Chapter
II, Section B.2.b of the HomVEE Draft Version 2 Handbook). When
determining which models meet HHS criteria for an ``evidence-based
early childhood home visiting service-delivery model,'' HomVEE
considers both whether the research that calculated the findings was
well designed, and whether the findings come from different studies
(with distinct samples). See Exhibit II.11 of the HomVEE Draft Version
2 Handbook for details.
2.2 Establish a 20-Year Moving Search Window for Reviewing Most
Manuscripts
Searches in HomVEE's first 11 annual reviews were for manuscripts
published in or after 1989. Generally, the HomVEE Draft Version 2
Handbook updates the description of how HomVEE identifies research.
Now, HomVEE proposes to implement a 20-year moving window for
previously unreviewed manuscripts to be eligible for review. Beginning
with the 2021 review, to keep the review current, HomVEE proposes
establishing a 20-year moving search window for previously unreviewed
manuscripts to be eligible for review. For example, for the 2021
review, HomVEE would consider manuscripts released or published in 2001
through 2020. However, HomVEE proposes that two categories of older
research remain eligible for review (1) older research that HomVEE has
already reviewed and (2) research submitted at any time (that is, since
HomVEE's inception and moving forward) through the call for research.
This change is described in Chapter II, Section A.1.a.1 of the HomVEE
Draft Version 2 Handbook.
2.3 Adopt the PRESS Method for Systematic Searching
Professional librarians have always conducted annual HomVEE
literature searches using a transparent process in which the databases
and search terms are published on the HomVEE website. Beginning with
the 2021 review, in recognition of accepted practice in the library
science field, HomVEE proposes to use a modified Peer Review of
Electronic Search Strategies (PRESS) method to refine the search terms
(McGowan et al. 2016). This approach includes adjusting search terms
and search databases in keeping with the recommendations of
professional librarians. This change is described in more depth in
Chapter II, Section A.1.a.1 of the HomVEE Draft Version 2 Handbook.
2.4 Add New ``Grey Literature'' Databases
To better capture research that is not published in academic
journals, HomVEE proposes to expand its annual search, beginning in
2021, to include two new databases to identify this ``grey
literature'': Google Scholar and the Harvard Think Tank Search. See
Chapter II, Section A.1.a.1 of the HomVEE Draft Version 2 Handbook.
2.5 Establish Rule for Accepting Supplemental Information
HomVEE will continue, as it always has, to accept submissions from
the public, both during its call for research and at other times, and
to consider those annually as part of prioritizing research for review.
The HomVEE Draft Version 2 Handbook proposes a clarification stating
that, if authors submit unpublished work to the HomVEE call for
research, HomVEE will consider only unpublished full manuscripts with
sufficient text describing the study's procedure, analysis approach,
and findings.
As part of the HomVEE Draft Version 2 Handbook (Chapter II, Section
B.1.b), HomVEE proposes establishing a new rule about accepting
supplemental information from stakeholders. Under the new rule, HomVEE
would accept supplemental information only under specific
circumstances. HomVEE must maintain a strict review schedule for the
annual review to ensure results are released on time.
Supplemental information can take two forms (1) new information
about a study's methods or procedures, or (2) new research that
supplements what HomVEE had on hand at the close of that year's call
for research, such as additional findings or new analyses of research
in a previously reviewed manuscript, or an entirely new set of
findings.
2.5.1 HomVEE Rule About New Information
HomVEE proposes to incorporate new information about methods and
procedures into the initial review of a manuscript only if (1) it is
provided in direct response to an author query and (2) authors submit
it in time for reviewers to examine it during the same annual review
cycle in which HomVEE issued the query. Otherwise, HomVEE intends to
require authors wait until HomVEE releases its annual review results
for the model described in the manuscript in question. Then, authors
could follow the process for requesting a reconsideration of evidence
to ask HomVEE to examine supplemental information that authors provide,
through the appeals process, about methods or procedure.
2.5.2 HomVEE Rule About New Research
HomVEE proposes to treat all new research as a submission to the
following year's call for research, unless it consists of new analyses
conducted at the explicit request of the HomVEE review team (see
Sections 3.3.1 and 3.3.2, below on repeated measures studies and
structural equation models, respectively).
2.6 Define Contrasts in Impact Research That Are Ineligible for Review
by HomVEE
To date, HomVEE has placed no restrictions on services offered to
the comparison condition in the impact studies it reviews. Beginning
with the 2021 review, HomVEE proposes that the review generally exclude
research that isolates the impact of model features. Research on
specific features does not answer HomVEE's core question of whether an
early childhood home
[[Page 47379]]
visiting model is effective. Specifically, HomVEE proposes not to
review studies about the impact of model features. However, studies
isolating the impact of a curriculum module may be treated as evidence
for an independent model if all of the following criteria are met:
The curriculum module satisfies the definition of an early
childhood home visiting model;
The treatment group does not receive any other curriculum
modules from the base model; and
The curriculum module has a manual and implementation
infrastructure independent from that of the base model.
This change is described in Chapter III, Section A.2 of the HomVEE
Draft Version 2 Handbook.
2.7 Defining Subgroups and Protocol for Reporting Subgroup Analyses
In the HomVEE Draft Version 2 Handbook, HomVEE defines a subgroup
as a subset of the sample that researchers choose to examine in a
study; that is, an analytic subgroup.
Subgroup research is important for HomVEE because a model can meet
the HHS criteria for ``evidence-based early childhood home visiting
service delivery models'' based on findings from subgroups. The HHS
criteria for an ``evidence-based early childhood home visiting service
delivery model'' include special rules about subgroup findings: If
favorable results that could form the evidence base for a model ``[are]
found for subgroups but not for the full sample for the study, [the
findings must] be replicated in the same domain in two or more studies
using non-overlapping analytic study samples.'' (See Exhibit II.11 in
the HomVEE Draft Version 2 Handbook.) Therefore, HomVEE exercises care
in identifying subgroup research and understanding how the subgroup
relates to the overall study sample.
HomVEE defines a subgroup as a subset of the overall sample
examined in a study--that is, an analytic subgroup (see Section 2.1,
above). Notably, this is different from defining subgroup as a subset
of the overall population. Although researchers may examine an analytic
subgroup in hopes of making inferences about a subset of the
population, the goal of the HHS criteria is to ensure that program
impacts are replicated consistently for an outcome domain. Such
replication is what gives HomVEE confidence that evidence of
effectiveness is not due simply to chance. Thus, if a model meets HHS
criteria for evidence of effectiveness based on subgroup findings, this
means that research in which that subgroup was similarly defined in
relation to the broader sample had consistent, favorable (statistically
significant) findings in distinct study samples.
Subgroup results may be nested within a manuscript (for example,
results from teenage mothers when the overall results in the manuscript
are from mothers with a range of ages), or they may be the main focus
of a manuscript (for example, a manuscript focusing on results from
teenage mothers when the overall study sample included mothers with a
range of ages). HomVEE treats both of those as analytic subgroup
analyses. HomVEE's definition means that not all analyses restricted to
a certain characteristic are subgroup analyses. For example, results
from teenage mothers are not an analytic subgroup analysis when the
overall study only enrolled teenage mothers, even though teenage
mothers are a subgroup of the population of mothers as a whole.
Because HomVEE's mission is to identify which models are effective
according to the HHS criteria, and to use project resources
judiciously, HomVEE proposes to only review research on replicable
subgroups (if it meets other eligibility criteria defined in Chapter
II, Section A of the HomVEE Draft Version 2 Handbook), and to only
report review results for replicated subgroups. HomVEE proposes the
following definitions for those terms:
Replicable subgroups are defined by a characteristic that
a different study could replicate with a non-overlapping sample. Most
subgroups are replicable, in theory. However, HomVEE does not consider
subgroups defined by cohort or time (for example, a subgroup of mothers
enrolled in 1995 in a study that included mothers enrolled across
several years) to be replicable in subsequent studies, and therefore
does not review time-based subgroups. Similarly, HomVEE will only
consider a subgroup defined by location to be replicable if the
location was selected based on defined characteristics (for example,
county with the highest teen birth rate in the state in a study
conducted in several counties). Location-based subgroups defined by a
location name (for example, Adams County in a study conducted in
several counties) will not be reviewed because the HomVEE team cannot
confidently verify whether the subgroup sample in a subsequent study in
that county overlaps with the first study when the team applies HHS
criteria.
HomVEE will report subgroup results only from a replicated
subgroup, one that has an identical definition in two non-overlapping
research samples. For example, a study examining a subgroup of
primiparous teenagers is not replicated by a study examining
primiparous women of all ages. This approach is consistent with the HHS
criteria's emphasis on observing effects across independent samples.
2.8 Clarify HomVEE's Approach to Operationalizing the HHS Criteria for
Randomized Controlled Trials
As specified in the statute that authorized the MIECHV Program and
required HHS to establish criteria for evidence of effectiveness of
home visiting models, the HHS criteria for an ``evidence-based early
childhood home visiting service delivery model'' state that additional
criteria apply when the research on home visiting models comes from
randomized controlled trials (see Exhibit II.11 in the HomVEE Draft
Version 2 Handbook). Specifically, one or more favorable impacts must
be sustained for at least 1 year after program enrollment, and one or
more favorable impacts must be reported in a peer-reviewed journal. The
HomVEE Draft Version 2 Handbook clarifies the way that HomVEE has
operationalized the additional criteria for randomized controlled
trials (RCTs). Specifically, these two requirements can be satisfied by
findings from different studies, provided the quality of these findings
is rated as moderate or high.
3.0 Clarifications and Changes to HomVEE Standards
HomVEE proposes several updates to its standards for reviewing
manuscripts about impact studies, including both clarifications and
changes.
It is natural in the course of a systematic review for issues to
arise that cannot be addressed by existing standards and rules. These
issues require the HomVEE team to develop internal guidance to guide
the review.
Clarifications to the standards added to the HomVEE Draft Version 2
Handbook, summarized in Section 3.1 below, represent HomVEE's attempt
to formalize internal guidance generated over the course of conducting
reviews so that the procedures are applied systematically.
Similarly, as research methods evolve, it is necessary for ongoing
systematic reviews to change their standards to meet the needs of the
field. Changes are HomVEE's attempt both to align with aspects of other
ongoing, federally sponsored systematic evidence reviews (Section 3.2)
and to specify standards for research designs that are becoming
increasingly common in home visiting studies (Section 3.3).
[[Page 47380]]
3.1 Clarifications
3.1.1 Changes to Terminology Used
HomVEE reviews manuscripts about research that uses any of three
types of quasi-experimental designs (QEDs)--regression discontinuity
designs (RDDs), single-case designs (SCDs), and non-experimental group
designs (NEDs). Previously, HomVEE used QED to refer only to NEDs.
Other designs, including SCD and RDD, are also quasi-experimental, so
HomVEE proposes labeling this category of research more precisely, as
NED (see Chapter I, Section C of the HomVEE Draft Version 2 Handbook).
HomVEE intends to use this new terminology to more accurately reflect
the fact that HomVEE does not have (and does not propose to implement)
requirements about statistical matching in NED designs.
3.1.2 Ineligible and Preferred Analyses
To date, as long as the underlying study design is an RCT or QED,
HomVEE has not specified rules for identifying analyses as ineligible
for review.
Starting with the 2021 review, HomVEE proposes to exclude certain
analyses within manuscripts about RCTs and QEDs as ineligible, as
described in Chapter III, Section A.3 of the HomVEE Draft Version 2
Handbook. HomVEE's mission is to determine whether research shows that
a home visiting model improves outcomes for children and families.
Questions about the mechanisms behind how a model works, the settings
where it might work best, and the populations who benefit the most from
the intervention are outside of the scope of the HomVEE review.
Although answers to these questions are important for understanding and
improving home visiting models, the primary aim of the HomVEE review is
to identify currently available models that are effective. For this
reason, certain types of analyses designed to answer questions other
than whether a model is effective are not eligible for review.
In addition, analyses of how the home visiting model affected only
sample members who received it are sometimes ineligible for review if
other analyses in the manuscript better address HomVEE's mission.
3.1.2.1 Mediating and Moderating Analyses
First, HomVEE proposes that most mediating and moderating analyses
(except some structural equation models, see Section 3.3.2 below),
would be ineligible for review. HomVEE focuses on research that answers
the following question: Is the home visiting model effective? Mediating
and moderating analyses answer important, but slightly different
questions of how, and for whom, the model works.
3.1.2.2 Endogenous Analyses
Second, HomVEE proposes that the review would exclude analyses that
control for endogenous characteristics. These characteristics (1) are
defined by behavior emerging after study participants know whether they
will be in the intervention group or the comparison group and (2) could
theoretically be affected by a home visiting model. Analyses that
control for endogenous characteristics produce biased estimates of the
effectiveness of an intervention. Analyses of subgroups defined by
endogenous characteristics would also be ineligible for review.
3.1.2.3 Analyses of the Impact of the Treatment on the Treated
The HomVEE Draft Version 2 Handbook also specifies a proposed
clarification to how HomVEE would review studies that examine the
effect of the treatment (the home visiting model) on the treated (study
sample members who receive the treatment). Specifically, when a study's
researchers examine the effect of both the intent to treat (ITT) and
the treatment on the treated (TOT), HomVEE proposes to focus its review
on the ITT, because those estimates more realistically depict the
average magnitude of the effect that a program replicating the model
would cause. If those researchers report only TOT estimates, HomVEE
reviews those using What Works Clearinghouse (WWC) Version 4.1 guidance
on reviewing for Complier Average Causal Effects.
3.1.3 Eligible Outcomes and Baseline Assessability
Since its inception, HomVEE has reviewed findings of home visiting
impact studies that fall into eight domains related to child, maternal,
and family well-being. In the HomVEE Draft Version 2 Handbook, HomVEE
proposes clarifying which specific findings within eligible analyses
are eligible for review (see Chapter III, Section A.4). These
clarifications formalize and expand HomVEE's existing internal guidance
on eligible outcomes and baseline assessability.
First, the HomVEE Draft Version 2 Handbook clarifies that only
unique findings would be eligible for review (those that report results
on a different outcome, sample or subgroup, or time period, or with a
different analytic approach, than findings reported in other
manuscripts about the same home visiting model). In these cases,
consistent with current HomVEE practices, the review simply would
reference the other manuscript--the first or most complete one in which
HomVEE encountered the finding--where HomVEE users could find those
results and the review conclusions. The Draft Version 2 Handbook also
clarifies that HomVEE would not consider simple transformations of
analyses with the same sample, outcome, and time period to be unique
findings within a manuscript if they (1) transform findings data from
frequency to a ratio (such as percentage or per thousand) or (2)
transform findings data across different ratio types (such as from
percentage to per thousand) because these simple transformations do not
constitute a different analytic approach. In manuscripts with such
transformations, HomVEE proposes to review the finding that is
calculated as a percentage, because it is an intuitive measure to many
readers and can be easily compared across studies.
Second, the HomVEE Draft Version 2 Handbook specifies
categorization practices and baseline equivalence requirements for
outcomes that HomVEE reviews. See Exhibit III.2 and Chapter III,
Section B.3 of the HomVEE Draft Version 2 Handbook for a summary and
Appendix C of that document for a detailed listing. That appendix
indicates which outcomes or outcome categories belong under each of the
eight domains. Also, although HomVEE requires NEDs and certain RCTs
(those with high attrition or compromised randomization) to establish
that the intervention and comparison groups are equivalent at baseline,
the review team recognizes that some measures cannot or should not be
measured at baseline. Therefore, Appendix C of the HomVEE Draft Version
2 Handbook clarifies which outcomes HomVEE would expect authors to
assess at baseline.
3.2 Changes To Align HomVEE With Standards of Other Federally Sponsored
Systematic Reviews
HomVEE's initial standards aligned to WWC standards, Version 2.1,
which were the latest standards implemented when the HomVEE review
began. These standards define the criteria that research must meet to
be assigned each of three ratings. HomVEE calls these ratings high,
moderate, and low, although the WWC rates research as Meets Standards
(HomVEE high), Meets Standards with Reservations (HomVEE moderate), and
Does Not Meet Standards (HomVEE low). The WWC remains a prominent and
influential
[[Page 47381]]
federally sponsored systematic evidence review. In early 2020, WWC
released Version 4.1 standards. Furthermore, in the time since HomVEE
began, ACF has begun overseeing another, related systematic review: The
Title IV-E Prevention Services Clearinghouse. That review focuses on
child welfare research, some of which overlaps with home visiting
research, and its standards are similar to those of WWC Version 4.1.
ACF is interested in aligning standards for HomVEE and the Prevention
Services Clearinghouse where appropriate.
In its HomVEE Draft Version 2 Handbook, HomVEE proposes to adopt
many aspects of the latest WWC standards and some aspects of the
Prevention Services Clearinghouse standards so that the review stays
synchronized with accepted best practices in federally sponsored
systematic reviews. The sections below describe proposed changes to the
HomVEE review that would affect study ratings as the HomVEE criteria
stand now. (HomVEE proposes to fully adopt WWC Version 4.1 criteria for
regression discontinuity design studies, which are not described below
because HomVEE has not, to date, reviewed any studies with this
design.)
3.2.1 Requirement for Validity and Reliability of Outcome Measures
To date, HomVEE has had no stated validity and reliability
requirements that outcomes must meet, although the review reports
whether outcomes are primary (which HomVEE defines as an outcome
measured through direct observation, direct assessment, or
administrative data; or self-reported data collected using a
standardized [normed] instrument) or secondary (for HomVEE, most self-
reported data, excluding self-reports based on a standardized
instrument). With the Draft Version 2 Handbook (see Chapter III,
Section B.4), HomVEE proposes to introduce face validity and
reliability standards. HomVEE reviewers will apply these new standards
to all findings that are within one of HomVEE's eight outcome domains
and to all measures HomVEE uses to assess baseline equivalence.
Findings about outcomes that do not meet both the face validity and the
reliability standard would rate low. With this change, HomVEE proposes
to stop sorting outcomes as primary or secondary.
To meet the face validity standard, an outcome measure must be (1)
clearly defined and (2) measure the construct it was designed to
measure. This information could come from the manuscript reviewers
examine, or from supplemental information that HomVEE requests from the
author. HomVEE reviewers propose to consult with project leaders
whenever it is not clear whether a measure meets the validity
requirement, and project leaders would in turn consult with subject
matter experts and with ACF about the validity of new and of modified
standardized measures.
Some outcome measures are not appropriate to validate with
psychometric tests. HomVEE proposes to assume that the following
measures are reliable: (1) Administrative records obtained from child
welfare or other social service agencies, hospitals or clinics, and
schools; (2) demographic characteristics; and (3) medical or physical
tests.
Otherwise, to demonstrate reliability, outcome measures must meet
at least one of the following standards:
Internal consistency (such as Cronbach's alpha) of 0.50 or
higher.
Test-retest reliability of 0.40 or higher.
Inter-rater reliability (as indicated by percentage
agreement, correlation, or kappa) of 0.50 or higher.
Under the proposed approach, HomVEE reviewers would prioritize
reliability statistics on the sample of participants in the manuscript
under review, but would also consider statistics from test manuals or
studies of the psychometric properties of the measures. The review team
may ask authors to provide additional information about the reliability
of their measures.
3.2.2 In Some Cases, Some Sample Loss Does Not Count as Attrition
Attrition happens when outcome data are missing for some members of
the intervention and comparison groups in a study. Previously, HomVEE
counted all sample loss as attrition unless the authors had imputed
findings (see Section 3.2.3, below). In alignment with Version 4.1 of
the WWC Standards, the HomVEE Draft Version 2 Handbook (see Chapter
III, Section B.1) proposes that some types of sample loss will not
count as attrition in HomVEE. First, losing sample members after random
assignment because of acts of nature, such as hurricanes, fires, or the
COVID-19 pandemic, is not considered attrition if the loss affects the
intervention and comparison conditions in the same way. However, if the
sample loss due to an act of nature was concentrated in one of the
conditions, then the sample loss would be considered attrition. Second,
when researchers exclude a subsample of the randomly assigned sample
from their analysis, HomVEE would not consider that excluded subsample
to constitute attrition if (1) the subsample was randomly selected or
(2) the subsampling was based on characteristics that were clearly
determined before the start of the intervention and applied
consistently across the intervention and comparison conditions.
3.2.3 Standards for Addressing Missing Data
The original HomVEE standards did not specify how reviewers would
respond when study authors used various analytic strategies to account
for missing data. The HomVEE Draft Version 2 Handbook (see Chapter III,
C.2, as well as Appendix F) proposes to align HomVEE's practices to the
way WWC Version 4.1 standards handle studies with missing data.
Specifically, HomVEE proposes to first calculate attrition based on the
analytic sample in the manuscript, treating any imputed values as lost
sample. If baseline data in the analytic sample are missing or imputed,
baseline equivalence would have to be established using the largest
baseline difference accounting for missing or imputed baseline data.
Second, manuscripts about studies with missing data would only be
eligible for review by HomVEE if the authors had used the following
specific approaches to address the missing data:
Complete case analysis
Maximum likelihood (including expectation maximization and
full information maximum likelihood)
Multiple imputation (must be conducted separately by treatment
status)
Nonresponse weights (must be conducted separately by treatment
status; acceptable only for missing outcome data, not for missing
baseline data)
In alignment with WWC version 4.1 standards, if the baseline data
include imputed data, HomVEE would also apply other criteria when
assessing baseline equivalence (see 3.2.5).
3.2.4 No Baseline Equivalence Requirement for Low-Attrition RCTs
Original HomVEE standards require authors of RCTs and non-
experimental designs to establish baseline equivalence on race and
ethnicity, socioeconomic status, and measures of outcomes that are
feasible to assess when the study begins.
In low-attrition RCTs, the original standards allow authors to
instead implement statistical controls for these characteristics. In
alignment with WWC Version 4.1 Standards, the HomVEE
[[Page 47382]]
Draft Version 2 Handbook proposes that HomVEE would no longer require
that RCTs with low attrition establish equivalence or adjust for
baseline differences. This is because proper randomization is expected
to produce groups that are similar, and baseline differences that might
be observed on one or more measures are not generally evidence of
differences that will introduce bias into research findings.
3.2.5 Baseline Equivalence Depends on Difference in Effect Sizes, and
Other Considerations
The original HomVEE standards based the assessment of equivalence
on measuring statistically significant differences between intervention
and comparison groups at baseline. In line with WWC Version 4.1
standards, HomVEE's Draft Version 2 Handbook proposes that HomVEE will
assess baseline equivalence based on the magnitude of the difference in
standard deviation units (effect size).
To limit bias that can arise from differences in the treatment and
comparison group units used to measure the effect of a home visiting
model on outcomes, the groups must appear similar on the relevant
baseline characteristics that are thought to be related to the
outcomes. This balance is best shown using the observed magnitude of
differences in the sample.
Specifically, the new HomVEE criterion for baseline equivalence
proposes to rely on effect size, computed as the absolute value of the
difference between treatment and comparison groups in standard
deviation units. HomVEE would require the following to be true for
research to demonstrate baseline equivalence for a specified
characteristic:
A baseline effect size less than or equal to 0.05 meets
the baseline equivalence requirement and requires no statistical
adjustment.
For a baseline effect size that is greater than 0.05 and
less than or equal to 0.25, an acceptable statistical adjustment for
the baseline characteristic is required to meet the baseline
equivalence requirement.
If the baseline effect size is greater than 0.25, HomVEE
considers the intervention and comparison groups to be nonequivalent,
that is, the intervention and comparison groups do not meet the
baseline equivalence requirement for the specified characteristic.
Under the proposed new standards, HomVEE would also consider the
following when assessing baseline equivalence:
HomVEE would allow baseline data that include imputed data
to be used to demonstrate baseline equivalence of the analytic sample
in some cases. If the baseline data include imputed data, HomVEE would
first estimate how large the baseline difference (in standard deviation
units) between intervention and comparison groups might be under
different assumptions about how the missing data are related to
measured and unmeasured factors. Then HomVEE would use the largest of
those estimates in absolute value as the effect size for assessing
baseline equivalence.
The measures used to establish baseline equivalence must
be at the same level as the unit of analysis. For example, in an
analysis at the individual or family level, measures of socioeconomic
status at the ZIP code level may not be used to establish baseline
equivalence between the individuals or families in the intervention and
comparison groups.
If the impact analyses use weights, then the baseline
means must be calculated using the same weights.
If the study conducted random assignment within blocks or
strata, and the analyses include dummy variables that differentiate
these blocks or strata, then these same dummy variables can be used to
adjust the baseline means.
This criterion is described in the HomVEE Draft Version 2 Handbook,
Chapter III, Section B.2.
3.2.6 Allowable Statistical Adjustment Techniques
HomVEE has always required that authors implement statistical
adjustments for baseline differences if their studies use (1) an RCT
design or (2) one type of quasi-experimental design, NED. To date,
HomVEE has not specified allowable techniques for that adjustment. With
the HomVEE Draft Version 2 Handbook (see Chapter III, Section B.2.b),
HomVEE proposes to follow WWC Version 4.1 guidelines about which
statistical adjustment procedures are acceptable. Those are:
Acceptable analytic methods to adjust for baseline
differences:
[cir] Regression adjustments
[cir] Analysis of covariance (ANCOVA) or multivariate analysis of
covariance (MANCOVA)
[cir] Repeated measures analysis of variance (ANOVA) or multivariate
analysis of variance (MANOVA)
[cir] Estimating impacts only for groups defined at baseline (for
example, ever had a baby versus never had a baby)
[cir] Growth curve modeling (this approach to modeling repeated
measures research is also subject to other requirements; see 3.3.1
below)
Acceptable methods if baseline and follow-up measures of
outcome are the same and have a strong relationship to each other
[cir] Gain or change scores (pre-post differences)
[cir] Difference-in-difference adjustments
[cir] Fixed effects for individuals
3.2.7 Cluster RCTs
HomVEE rarely encounters RCTs with a cluster design in which a
group of sample members, such as a neighborhood, is assigned to be
offered a home visiting model or some other condition. However, in such
designs, limiting sample loss at both the cluster (for example, a
neighborhood) and subcluster (for example, a family that received home
visiting) levels is important to maintaining the integrity of the
randomization design. Under HomVEE's original standards, reviewers
would apply a cluster correction to findings if authors themselves had
not done so, but no special requirements were in place for rating
studies that used a cluster design.
Instead, in research reviewed under the HomVEE Draft Version 2
Handbook, HomVEE proposes to align to WWC Version 4.1 guidelines for
calculating attrition and non-response of subcluster members (such as
individuals or families) in cluster RCTs. Exhibit III.14 of the HomVEE
Draft Version 2 Handbook specifies how studies would be rated based on
their combination of attrition at the cluster and individual levels,
and authors' decisions about implementing statistical controls. In
brief, a cluster RCT would be eligible to rate high only if it has low
sample loss at both the cluster level and the individual level. To rate
moderate, research about cluster RCTs with high attrition and research
about cluster NEDs would need to demonstrate baseline equivalence of
the analytic sample. Additional detail about this new standard appears
in Chapter III, Section C.1 of the HomVEE Draft Version 2 Handbook.
3.2.8 Adopt New WWC Version 4.1 Standards for Regression Discontinuity
Designs
Regression discontinuity research has been eligible for review by
HomVEE since the project's inception, using earlier WWC pilot criteria
for this research. This design is rare in home visiting research, and,
to date, HomVEE has not reviewed any research that uses this design.
HomVEE proposes to align its Version 2 standards to WWC's latest
(Version 4.1) RDD standards. The updates to the RDD standards consist
of:
[[Page 47383]]
A new set of procedures for reviewing ``fuzzy'' RDDs (for
example, those in which some intervention group members do not receive
intervention services and the analysis adjusts for this
nonparticipation),
Expanded procedures for reviewing multi-site and multiple
assignment variable regression discontinuity designs, and
A preference for local bandwidth impact estimation over
global impact regression with flexible functional forms.
Appendix D of the HomVEE Draft Version 2 Handbook thoroughly
describes the new WWC Version 4.1 RDD standards that HomVEE propose to
implement, and their corresponding reporting procedures.
3.3 Other Changes
The HomVEE Draft Version 2 Handbook proposes two other changes to
standards for reviewing impact studies. The two changes would define a
new approach to reviewing designs that are becoming increasingly common
in home visiting studies--repeated measures analyses and structural
equation models. The third change pertains to SCD research.
3.3.1 Repeated Measures Analyses
In repeated measures analyses, authors measure the research sample
at several time points to chart its growth over the course of the
intervention and sometimes beyond. To date, HomVEE has not specified
any standards for reviewing repeated measures analyses in group-design
studies (such as RCTs and NEDs), nor have other federally sponsored
systematic evidence reviews thoroughly addressed this.
In the proposed new standard, HomVEE would only review and report
findings from repeated measures analyses with multiple follow-ups in
RCTs and NEDs when the findings are available for individual time
points, relative to baseline. When rating each time point, HomVEE would
apply its Version 2 RCT or NED standards. Generally, when gathering
information to rate each time point, HomVEE would defer to what the
author reported or what the review team could calculate based on
details the author provided. As a last resort, when adjusted analyses
are necessary in order to rate the study, and the HomVEE team cannot
make the necessary calculations, the HomVEE team would ask authors to
reanalyze their data to calculate adjusted time point findings. HomVEE
would exclude from its review of a repeated measures analysis any time
points for which an impact cannot be included in HomVEE reports because
neither author-provided nor HomVEE-calculated estimates are available.
Chapter III, Section C.3 of the HomVEE Draft Version 2 Handbook
describes the proposed new approach to rating repeated measures
analyses in detail.
3.3.2 Structural Equation Models
Structural equation models (SEMs) examine the relationship between
a dependent variable and multiple independent variables, often
incorporating multiple outcomes from different follow-up periods. To
date, HomVEE standards did not define how the review would incorporate
SEM research. Chapter III, Section C.4 of the HomVEE Draft Version 2
Handbook specifies how HomVEE proposes to approach the review of
research with this design going forward. In brief, only SEMs that are
accompanied by a path diagram (including one authors may submit in
response to a query from HomVEE) and that are identified (that is, the
degrees of freedom are greater than the parameters to be estimated)
would be eligible for review. Within SEMs that are eligible for review,
HomVEE would review only findings for which the answer to the following
two questions is yes: (1) Is there a direct pathway from the
intervention to the outcome? and (2) Are there no pathways leading to
that outcome from another outcome? This approach is consistent with
HomVEE's proposed new approach to mediated and moderated analyses. See
also Section 3.1.2.1 above.
3.3.3 Review of Single-Case Design (SCD) Research
SCDs are quasi-experimental research designs in which an individual
case serves as its own control, and the outcome is measured repeatedly
within and across different conditions (as defined in What Works
Clearinghouse [WWC] Version 4.1 standards). SCD research has been
eligible for HomVEE review since its inception, using earlier WWC pilot
criteria for this research. With Version 4.1, WWC has removed the
``pilot'' designation from its standards and has updated its procedures
for reviewing SCD research in several ways; HomVEE proposes aligning to
WWC's version 4.1. Although WWC previously standards instructed
reviewers to only use visual analysis of changes in the outcome over
time and across conditions to characterize the findings from an SCD
study, the new standards from WWC also have reviewers calculate and use
a design-comparable effect size to characterize the findings. Reviewers
would still use visual analysis to assess whether a SCD study is well
designed. To calculate a design-comparable effect size, the HomVEE
contractor review team would use data presented in the study if
possible, or (only if necessary) contact the study authors to request
raw study data so the team could calculate that value. Appendix E of
the HomVEE Draft Version 2 Handbook thoroughly describes the WWC
Version 4.1 SCD standards that HomVEE propose to implement, and their
newly updated reporting procedures.
4.0 Timeline for HomVEE To Apply New Procedures and Standards
HomVEE proposes to apply the new procedures and standards beginning
with the 2021 review. HomVEE will not retroactively apply the new
standards to previously reviewed research about evidence-based models
unless it is SCD research about a model HomVEE prioritizes and selects
for review.
To promote consistency in reporting across the review,
clarifications about the outcomes that are eligible for review in each
domain retroactively will apply to all models regardless of (1) their
evidence-based status according to HHS criteria and (2) whether they
are prioritized and selected for review. However, manuscripts that have
findings excluded or moved to other domains will not be re-reviewed
with HomVEE Version 2 standards (unless they are manuscripts about a
SCD study). In addition, the HomVEE team will retroactively apply
clarified definitions of study, manuscript, and subgroup, with the aim
of relabeling HomVEE products so they use consistent language.
Also, HomVEE typically reviews eligible models every other year at
the earliest. In 2021 (the first year that new procedures and standards
are in effect), HomVEE will suspend this rule for one year only, so
that models reviewed in 2020 are not excluded from consideration for
the 2021 review.
4.1 HomVEE Will Not Retroactively Apply New Procedures and Standards to
Inactive Models
For models that no longer provide implementation support, the
HomVEE team generally does not plan to retroactively apply the new
procedures and standards, except to apply the clarifications about the
outcomes that are eligible for review in each domain and about the
definitions of study, manuscript, and subgroup. The team proposes to
update reports about those models on the HomVEE website (https://homvee.acf.hhs.gov) to indicate that they were reviewed under the
[[Page 47384]]
initial HomVEE procedures and standards.
5.0 Request for Information (RFI)
Through this Federal Register Notice, ACF is soliciting information
from a broad array of stakeholders on the proposed revisions to
HomVEE's procedures. Federal, state, and local decision makers rely on
HomVEE to know which home visiting models are effective. New
definitions, rules, and procedures about model versions may affect
which models are deemed effective by HomVEE. New procedures may affect
which models are eligible for review and deemed effective by HomVEE.
New standards may affect which studies constitute well-designed
research that serves as an evidence base for models that meet HHS
criteria for an ``evidence-based early childhood home visiting service
delivery model.''
Responses to this Federal Register notice will inform ACF's ongoing
discussion about HomVEE's procedures and standards, with the aim of
publishing a final HomVEE Version 2 Handbook by the end of 2020. This
RFI is for information and planning purposes only and should not be
construed as a solicitation or as an obligation on the part of ACF or
HHS.
(Authority: Social Security Act Title V Sec. 511 [42 U.S.C. 711],
as extended by the Bipartisan Budget Act of 2018 (Pub. L. 115-123)
through fiscal year 2022)
John M. Sweet Jr,
ACF/OPRE Certifying Officer.
References
U.S. Department of Education, Institute of Education Sciences,
National Center for Education Evaluation and Regional Assistance,
What Works Clearinghouse. What Works Clearinghouse Version 2.1
Evidence Standards. 2011. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v2_1_standards_handbook.pdf.
Accessed June 19, 2020.
U.S. Department of Education, Institute of Education Sciences,
National Center for Education Evaluation and Regional Assistance,
What Works Clearinghouse. What Works Clearinghouse Procedures
Handbook: Version 4.1., 2020a. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf.
Accessed June 19, 2020.
U.S. Department of Education, Institute of Education Sciences,
National Center for Education Evaluation and Regional Assistance,
What Works Clearinghouse. What Works Clearinghouse Standards
Handbook: Version 4.1., 2020b. Available at https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1-508.pdf.
Accessed June 19, 2020.
[FR Doc. 2020-17001 Filed 8-4-20; 8:45 am]
BILLING CODE 4184-74-P
