Agency Information Collection Activities: Comment Request; Merit Review Survey-2021 and 2023 Assessment of Applicant and Reviewer Experiences
| Published date | 03 February 2021 |
| Citation | 86 FR 8045 |
| Record Number | 2021-02240 |
| Section | Notices |
| Court | National Science Foundation |
Federal Register, Volume 86 Issue 21 (Wednesday, February 3, 2021)
[Federal Register Volume 86, Number 21 (Wednesday, February 3, 2021)]
[Notices]
[Pages 8045-8047]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2021-02240]
=======================================================================
-----------------------------------------------------------------------
NATIONAL SCIENCE FOUNDATION
Agency Information Collection Activities: Comment Request; Merit
Review Survey--2021 and 2023 Assessment of Applicant and Reviewer
Experiences
AGENCY: National Science Foundation.
ACTION: Submission for OMB review; comment request.
-----------------------------------------------------------------------
SUMMARY: The National Science Foundation (NSF) has submitted the
[[Page 8046]]
following information collection requirement to OMB for review and
clearance under the Paperwork Reduction Act of 1995. This is the second
notice for public comment; the first was published in the Federal
Register, and no comments were received. NSF is forwarding the proposed
submission to the Office of Management and Budget (OMB) for clearance
simultaneously with the publication of this second notice.
DATES: Written comments and recommendations for the proposed
information collection should be sent within 30 days of publication of
this notice to www.reginfo.gov/public/do/PRAmain. Find this particular
information collection by selecting ``Currently under 30-day Review--
Open for Public Comments'' or by using the search function.
FOR FURTHER INFORMATION CONTACT: Suzanne H. Plimpton, Reports Clearance
Officer, National Science Foundation, 2415 Eisenhower Avenue,
Alexandria, VA 22314, or send email to [email protected]. Individuals
who use a telecommunications device for the deaf (TDD) may call the
Federal Information Relay Service (FIRS) at 1-800-877-8339, which is
accessible 24 hours a day, 7 days a week, 365 days a year (including
federal holidays).
Copies of the submission may be obtained by calling 703-292-7556.
SUPPLEMENTARY INFORMATION: NSF may not conduct or sponsor a collection
of information unless the collection of information displays a
currently valid OMB control number and the agency informs potential
persons who are to respond to the collection of information that such
persons are not required to respond to the collection of information
unless it displays a currently valid OMB control number.
Title of Collection: Merit Review Survey--2021 & 2023 Assessment of
Applicant and Reviewer Experiences.
OMB Number: 3145-NEW.
Type of Request: Request for approval to establish an information
collection.
Proposed Project: The National Science Foundation (NSF) receives
close to 50,000 proposals for funding annually, each of which undergoes
a rigorous merit review process that is designed to ensure all
proposals are fairly and thoroughly reviewed. The merit review process
comprises three phases:
1. NSF announces funding opportunities on the NSF website and
Grants.gov. Applicants prepare proposals in response to these
opportunities and submit their proposals via FastLane (NSF's web-based
system for proposal submission and review) or Grants.gov.
2. Proposals are assigned to the appropriate program(s) for review.
Each proposal is assigned a Program Officer (PO) who selects external
reviewers to evaluate the proposal according to the two NSF merit
review criteria, Intellectual Merit and Broader Impacts. The
Intellectual Merit criterion encompasses the potential to advance
knowledge. The Broader Impacts criterion encompasses the potential to
benefit society and contribute to the achievement of specific, desired
societal outcomes. Programs may have additional review criteria
particular to the goals and objectives of the program. The NSF
guidelines for the selection of reviewers are designed to ensure
selection of experts who can give program officers the proper
information needed to make a recommendation in accordance with the
merit review criteria. POs utilize the proposal's reference list, the
investigator's suggested reviewers, and personal knowledge of
individual reviewers to identify a pool of diverse experts with respect
to type of organization represented, demographics, experience, and
geographic balance, selecting appropriate reviewers with no apparent
potential conflicts. Most proposals are reviewed by three to ten
content expert reviewers who provide written feedback on the proposal
through FastLane. POs synthesize reviewer comments and issue a
recommendation to either decline or award funding based on reviewer
feedback, panel discussions, the amount of available funding, and
portfolio balances (i.e., the diversity of a portfolio, including
factors such as award type, career stage, demographic characteristics,
geographic location, institution type, research topic, laboratory
funding status, and intellectual risk). The proposal and PO
recommendation is then forwarded to the appropriate Division Director
or other NSF official for additional review and action to either
decline or award.
3. Each proposal recommended for award undergoes an administrative
review conducted by NSF's Office of Budget, Finance, and Award
Management. If it passes this review, the proposal is awarded.
Through this review process, NSF aims to identify the highest
quality proposals to receive funding. The success of this process
hinges on the assumptions that applicants will continue to submit to
NSF their ideas for cutting-edge research and that experts in their
respective fields will continue to provide high-quality reviews of
those proposals.
The goal of this data collection is to assess the experiences of
applicants and reviewers and their satisfaction with NSF's merit review
process. The data collection for which this OMB approval is requested
includes a Web-based survey that will be administered to all applicants
and reviewers who participated in the merit review process between
fiscal years (FY) 2018 and FY 2020 (2021 survey) and between FY 2020
and FY 2022 (2023 survey).
The specific research objectives are to--
1. Assess applicant and reviewer perceptions of, and satisfaction
with, various aspects of the merit review process.
2. Document the time burden the merit review process places on
reviewers and applicants.
3. Examine applicant and reviewer perceptions of the quality of
reviews and of proposals.
4. Assess the changes in applicant and reviewer perceptions of
burden, satisfaction, and quality between the 2019 and 2021 surveys and
the 2021 and 2023 surveys.
5. Examine the variation of applicant and reviewer perception of
satisfaction, burden, and quality by key population subgroups,
including race/ethnicity, gender, and disability.
6. Describe the extent to which NSF's reviewer orientation video is
correlated with awareness of different types of cognitive biases and
the use of strategies to reduce cognitive bias and to provide
constructive feedback.
7. Describe the extent to which the elimination of annual proposal
deadlines affected reviewer and applicant burden, perceptions of
proposal and review quality, and satisfaction with the merit review
process.
8. Describe applicants and reviewers experiences with student
support programs as well as what NSF application and funding support is
associated with the receipt of financial support from NSF as an
undergraduate or graduate student.
Data from the survey will be used to improve NSF's implementation
of the merit review process.
Use of the information: The primary purpose of collecting this
information is program evaluation. The data collected will enable NSF
to assess the satisfaction, including perceptions of burden and
quality, of applicants and reviewers who participate in the merit
review process in order to monitor and improve the program and assess
its implementation. Findings will inform continual improvement
activities related to the merit review process.
[[Page 8047]]
Expected respondents: All applicants who have submitted proposals
and reviewers who have reviewed NSF proposals between FY 2018 and 2020
will be invited to participate in the 2021 survey and comparable
individuals who participated between FY 2020 and FY 2022 will be
invited to participate in the 2023 survey. This is estimated to be
approximately 87,000 individuals per survey round.
Average time per reporting: The online survey is comprised
primarily of close-ended questions and is designed to be completed by
respondents in approximately 20 minutes.
Frequency: Eligible applicants and reviewers will be asked to the
complete the 2021 Merit Review survey one time in fall 2021. For the
2023 survey, eligible applicants and reviewers will be asked to
complete the survey one time in fall 2023.
Estimate of burden: It is estimated the survey will require
approximately 20 minutes (on average) to complete. The anticipated
universe size for each survey cycle is 87,000 individuals, which
includes all applicants who submitted proposals and all reviewers
between FY 2018 and FY 2020 (for the 2021 survey) and between FY 2020
and FY 2022 (for the 2023 survey). The estimated survey response rate
for each the 2021 and 2023 survey rounds is 40 percent. Therefore, the
total burden is 23,200 hours; this is a respondent burden of 11,600
hours per survey year (2021 and 2023).
Based on 2019 merit review survey data, it is anticipated that most
survey respondents will be working at an academic institution, likely
in a teaching and/or research capacity. Therefore, for the purpose of
burden estimates, we have used the annual mean wage for postsecondary
teachers from Bureau of Labor Statistics, which is $79,540.\1\ Assuming
a 40-hour workweek over the course of 52 weeks annually, the hourly
wage for this occupation is approximately $38.00. Therefore, the
overall cost to survey respondents for each survey year (2021 and 2023)
would be approximately $440,800 (11,600 burden hours x $38.00 per
hour), as shown in table A.12.1 below.
---------------------------------------------------------------------------
\1\ Source: https://www.bls.gov/ooh/education-training-and-library/postsecondary-teachers.htm.
Table A.12.1--Estimate of Respondent Burden and Cost by Year
--------------------------------------------------------------------------------------------------------------------------------------------------------
Number of Average burden
Year Number of responses per per response Total burden Average Total cost
respondents respondent (hours) hours hourly wage
--------------------------------------------------------------------------------------------------------------------------------------------------------
2021...................................................... 34,800 1 0.33333 11,600 $38 $440,800
2022...................................................... 0 0 0 0 0 0
2023...................................................... 34,800 1 0.33333 11,600 38 440,800
---------------------------------------------------------------------------------------------
Total................................................. 69,600 1 0.33333 23,200 38 881,600
--------------------------------------------------------------------------------------------------------------------------------------------------------
Comments: Comments are invited on (a) whether the proposed
collection of information is necessary for the proper performance of
the functions of the Agency, including whether the information shall
have practical utility; (b) the accuracy of the Agency's estimate of
the burden of the proposed collection of information; (c) ways to
enhance the quality, utility, and clarity of the information on
respondents, including through the use of automated collection
techniques or other forms of information technology; and (d) ways to
minimize the burden of the collection of information on those who are
to respond, including through the use of appropriate automated,
electronic, mechanical, or other technological collection techniques or
other forms of information technology.
Dated: January 29, 2021.
Suzanne H. Plimpton,
Reports Clearance Officer, National Science Foundation.
[FR Doc. 2021-02240 Filed 2-2-21; 8:45 am]
BILLING CODE 7555-01-P
