APPENDIX A: RESEARCH METHODOLOGY

The sections that follow describe the research methodology used for the National Science Foundation Principal Investigator FY 2001 Grant Award Survey and for the National Science Foundation Institutional FY 2001 Grant Award Survey.

A. QUESTIONNAIRE DEVELOPMENT

The initial phase of questionnaire development included two focus groups with NSF representatives who could identify key issues to be included in the two questionnaires. A third focus group with institutional representatives was scheduled for September 2001, however the events of September 11 resulted in a cancellation. Instead institutional representatives were contacted by telephone to discuss key issues to be included in the survey. After draft questionnaires were developed, they were cognitively pretested with PIs and institutional representative, and revisions were made based on the findings from the pretests. The following provides details about the steps that were followed:

Date Type of Group Number of Participants

August 8, 2001 NSF Focus Group 12

August 9, 2001 NSF Focus Group 11

October 2001 Institutional Representatives 4

(Telephone interviews)*

December 4, 2001 Principal Investigators

Cognitive pretest/group discussion 8

January/February Institutional Representatives 4

2002 Cognitive pretest/individual

interviews

*Re-scheduled from the Federal Demonstration Project Group discussion because of September 11,2001.

TABLE B.1

SAMPLE ALLOCATION AND STRATA FOR INSTITUTION SAMPLE

	Number of Institutions
Strata	Sample Size	Equal Size Strata	Square Root Algorithm	Cube Root Algorithm
1	20	116	269	197
2	20	116	154	159
3	20	116	79	106
4	20	117	47	70
5	20	117	33	50
Half-Width of 95% Confidence Interval		0.098	0.115	0.100

Source: Mathematica computations.

Note: Half-width of 95% confidence interval = 1.96 * %variance for a stratified random sample where the variance within a stratum is computed from p * (1 – p) with p =0.50.

D. DATA COLLECTION

The PI survey was conducted using a mixed-mode format of Web and mail methods and the institution survey was a mail survey. A database containing contact information (telephone numbers and e-mail addresses) for potential respondents was provided to MPR by NSF.

The following provides additional detail of the data collection steps that were taken:

January 2001 NSF Director Dr. Rita R. Colwell sends PIs e-mail message announcing the survey.

January 30, 2002 MPR begins sending PI e-mail invitations with Web site access username and password on a rolling schedule.

February 4-19, 2002 MPR sends e-mail remiders to non-responders on a 3 day schedule.

February 15, 2002 MPR sends questionnaire mail packets to 778 PIs who have responded to the Web questionnaire.

March 8, 2002 Deadline for data collection

CONSTRUCTED VARIABLES

CALCULATION AND DATA SOURCE

Option 1: Award Efficiency and Effectiveness

Deviation from Requested Award Amount

(FY 2001 Award Request – FY 2001 Award

Amount)/Number of FY2001 Grant Award

Years

(Information from NSF data file)

Option 2: Award Efficient and Effectiveness

Percent of Research Being Funded

(FY 2001 Award Amount/(Q3.2¸100)-FY2001 Award Amount)

Divided by 5 Years to annualize

(NSF information and survey question)

Option 4: Award Efficient and Effectiveness

NFS’s Contribution

Q3.3 X Q3.4 Divided by 5 Years to Annualize

(Survey questions)

Difference in FY 2001 Award Amount

Request and Amount Awarded

FY 2001 Amount Request-FY 2001 Amount

Award

(NSF data file)

Difference in FY 2001 Duration Request and

Duration Award

FY 2001 Duration Request-FY 2001 Duration

Award

(NSF data file)

Additional Duration Needed

FY 2001 Duration Award + Q3.1

(NSF data file and survey question)

G. SURVEY MEASUREMENT ERROR

It should be noted that in any survey there are sources of both sampling and non-sampling error. Some examples of sources of survey measurement error are non-response to the survey, skipped questions, context effects, data collection methodology, and question wording. In conducting this study, all efforts possible were taken to minimize survey measurement error.

Text Box: Conducted for NSF by:

Text Box: Please Complete and Submit Questionnaire
By March 8, 2002

Text Box: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number of this project is 3145-0185.

OMB Approval Number: 3145-0185

Welcome to the

National Science Foundation

Principal Investigator

2001 Grant Award Survey

BY MAIL

Questions or Comments?

TO:

Matt Mishkind

Project Director

Mathematica Policy Research, Inc.

P.O. Box 2393

Princeton, NJ 08543

Contact Matt Mishkind at

877-236-4185

E-mail: nsfgrantsweb@mathematica-mpr.com

2001 NSF GRANT INFORMATION

#1	Grant Title	__________________________________
#2	Grant Effective Date	__________________________________
#3	Requested Amount	__________________________________
#4	Awarded Amount	__________________________________
#5	Amount Change 5% or Greater	__________________________________
#6	Requested Duration	__________________________________
#7	Awarded Duration	__________________________________
#8	Duration Change 1 Year or Greater	__________________________________

· You will be asked to reference the information listed above throughout this questionnaire. This information is from our database and is specific to the NSF grant you were awarded funding in 2001.

· When a question asks you to think about any of the above information, a notation will be made in the questionnaire. Therefore, it is important to keep this information attached to the rest of the questionnaire.

· If this is your grant, please check the box and begin the questionnaire.

· If any of this grant information is incorrect, please contact Matt Mishkind at 877-236-4185 or nsfgrantsweb@mathematica-mpr.com before you complete the questionnaire.

· You may also complete this questionnaire on the Web:

Log onto

http://nsfgrants.mathematica-mpr.com

and enter the following

USERNAME: xxxxxx

PASSWORD: xxxxxx

PRINCIPAL INVESTIGATOR

2001 GRANT AWARD SURVEY

SECTION 1

REMINDER: Please check grant information provided on back of cover page.

1.1 Was your 2001 NSF grant [#1 GRANT TITLE] awarded on [#2 GRANT EFFECTIVE DATE] a first‑time submission or a revision of a previously declined NSF proposal?

· A revised proposal does not refer to changes made in your 2001 NSF grant proposal after the initial review

mark one

71% a first time submission

29% a revision of a previously declined NSF proposal

1.2 NSF research grants can be classified along a number of different dimensions. Which ONE of the following definitions best describes the research that is funded by this grant?

· If your work involves several of these categories please choose the one that is most appropriate

THEORETICAL research can be accomplished with minimal physical resources beyond the investigator’s institutional research library, computing capability and office space.

LABORATORY research requires an equipped laboratory, for example, research often found in chemistry, biology or engineering university laboratories requiring research and/or testing equipment, plumbing.

FIELD research requires fieldwork, specimen collection, sample survey, location of sensors, etc. away from the principal investigator’s institution, for example, some science activities in geosciences, biology, social sciences.

mark one

37% Theoretical Research

44% Laboratory Research

18% Field Research

1.3 Does your 2001 NSF project require the use of a national or international research facility such as access to an accelerator, a light source, a ship, major telescope or supercomputer center?

16% Yes

83% No

1.4 In general, would you say that this 2001 NSF grant is funding:

mark one

7% A specific product or deliverable

89% A project that is part of your ongoing body of research and educational activities

4% Other (Please Describe)

1.5 For each of the following, how much advice did you get from NSF staff when you were preparing your grant proposal:

		MARK ONE FOR EACH
		A Great Deal	Some	Not Much	None At All
a.	The amount of funding.............	12%	27%	17%	43%
b.	The duration of the grant proposal.................................	11%	21%	16%	51%
c.	The substance or focus of the grant......................................	7%	25%	19%	49%

1.6 Based on the advice provided by NSF staff, did you increase, not change, or decrease:

		Increase		Not Change	Decrease	Not Asked
a.	The amount of the award you proposed.......................................		4%	36%	27%	31%
b.	The award duration you proposed....		3%	58%	6%	31%

SECTION 2

As part of the review process, NSF may ask principal investigators to revise their proposal before they are awarded funding. The following questions are about your revised budget and award duration.

ONLY ANSWER Q2.1 IF #5 AMOUNT CHANGE >5% IS LABELED “YES.” See inside cover.

2.1 In your proposal, you requested [#3 REQUESTED AMOUNT] and in your award you received [#4 AWARDED AMOUNT].

Overall, how much will this change in the award amount impact your ability to do what you expected to accomplish with this 2001 NSF grant?

mark one

1% Can do a great deal more than expected

2% Can do somewhat more than expected

7% Can do about the same as expected

28% Can do somewhat less than expected

15% Can do a great deal less than expected

1% Don’t know

47% Not asked

ONLY ANSWER Q2.2 IF #8 DURATION CHANGE >1 YEAR IS LABELED “YES.” See inside cover.

(IF BOTH #5 AMOUNT CHANGE >5% AND #8 DURATION CHANGE >1 YEAR ARE LABELED “NO,” PLEASE SKIP TO SECTION 3).

2.2 In your proposal, you requested [#6 REQUESTED DURATION] and in your award you received [#7 AWARDED DURATION].

Overall, how much will this change in award duration impact your ability to do what you expected to accomplish with this 2001 NSF grant?

mark one

1% Can do a great deal more than expected

1% Can do somewhat more than expected

1% Can do about the same as expected

4% Can do somewhat less than expected

5% Can do a great deal less than expected

88% Not asked

IF YOU RESPONDED AS 6 “CAN’T ANSWER” OR -1 “DON’T KNOW” TO BOTH Q2.1 AND Q2.2, PLEASE SKIP TO SECTION 3.

ONLY ANSWER Q2.3 IF YOU PROVIDED A RESPONSE OF 1, 2, 3, 4, OR 5 TO EITHER Q2.1 OR Q2.2.

2.3 The following are some possible consequences of the changes in your NSF award funding and/or duration. Will this change have a positive impact, no impact, or negative impact on your ability to ...

		Positive Impact	No Impact	Negative Impact	Not Applicable	Not Asked
A. Goals and Objectives
a.	Pursue innovative ideas.............................................................	4%	25%	23%	1%	47%
b.	Pursue high-risk ideas..............................................................	3%	20%	26%	4%	47%
c.	Obtain other funding.................................................................	6%	35%	6%	5%	47%
B. Applications and Outcomes
d.	Disseminate research findings...................................................	4%	32%	17%	1%	47%
e.	Develop instrumentation or other enhancements for the research and education infrastructure......................................................	2%	17%	20%	13%	47%
f.	Develop partnerships with industry, other educational institutions, or national laboratories..............................................................	3%	26%	13%	11%	47%
g.	Integrate research activity into your teaching and training.............	4%	27%	18%	4%	47%
h.	Nurture connections between research activity and its potential for: health benefits, economic benefits, and national security benefits...................................................................................	2%	24%	10%	16%	47%
i.	Develop programs with K‑12 teachers and/or students.................	1%	23%	7%	22%	47%
j.	Improve public understanding of the project.................................	3%	31%	11%	9%	47%
C. Process and Team Building
k.	Collaborate with researchers in your area of research..................	5%	21%	26%	1%	47%
l.	Broaden participation of under-represented groups in the research activity....................................................................................	3%	27%	18%	6%	47%
m.	Collaborate with researchers in different areas of research............	4%	25%	21%	3%	47%
n.	Achieve the research objectives within the specified time.............	4%	14%	34%	1%	47%
o.	Obtain quality personnel...........................................................	3%	17%	28%	4%	47%
p.	Establish mentoring or other research-based education activities..	3%	23%	21%	5%	47%
D. Research Tools							4%	47%
q.	Access state-of-the-art equipment.............................................	2%	28%	17%	6%	47%
r.	Access facilities.......................................................................	2%	34%	10%	6%	47%

SKIP Q2.4a IF NO POSITIVE ITEMS IN Q2.3

2.4a Among the items you marked “Positive Impact,” please rank order (write in the number(s)), up to three, those that had the most positive impact.

SKIP Q2.4b IF NO NEGATIVE ITEMS IN Q2.3

2.4b Among the items you marked “Negative Impact,” please rank order (write in the number(s)), up to three, those that had the most negative impact.

2.5 Please describe any other impact(s) that resulted from the change in your 2001 NSF award or give more details on any in the list that need further explanation.

SECTION 3

The next group of questions is your assessment of how this grant fits into your ongoing body of research and educational activities.

· Our records indicate that your 2001 NSF grant is for $[#4 AWARDED AMOUNT] over a period of [#7 AWARDED DURATION] Years. See inside cover.

3.1 Thinking about the timeframe for your ongoing body of research and educational activities, about how many additional years do you think you would need to accomplish your key goals?

· DO NOT include the years for the 2001 NSF grant

· Enter “0” for “Do not need any additional years”

Median=2

Mean=3

Mode=2

Range: 0 to 40

3.2 If you think about your ongoing body of research and educational activities as 100 percent of what you’d like to accomplish in the next five years, about what percent of what you’d like to do will be achieved with your 2001 NSF research grant?

Median=30

Mean=37

Mode=20

Range: 0 to 100

Now, speculate on what changes, if any, you would need to accomplish all you would like to in the next five years.

3.3 In the next five years, how much additional funding from all sources, if any, would you need to achieve what you would like to with your ongoing body of research and educational activities?

· Exclude funding you currently have for this NSF grant and from any other funding sources

· Enter “0” for “Do not need any additional funding”

Median=$500,000

Mean=$1,149,000

Mode=$500,000

Range: $0 to $300,000,000

IF YOU DO NOT NEED ADDITIONAL FUNDING OR DON’T KNOW, SKIP TO Q3.6.

3.4 What percent of this additional amount do you think is appropriate for NSF to fund?

Median=70%

Mean=67%

Mode=100%

Range: 0% to 100%

3.5 About how many additional grants do you think you would need to get this funding?

Median=2

Mean=2.39

Mode=2

Range: 0 to 32

ONLY ANSWER Q3.6 IF YOU NEED ADDITIONAL YEARS (Q3.1) AND/OR ADDITIONAL FUNDING (Q3.3).

IF YOU RESPONDED “0” OR “DON’T KNOW” TO Q3.1 AND Q3.3, SKIP TO SECTION 4.

3.6 If NSF provided this additional funding and/or duration to support your ongoing research and educational activities, would there be a positive impact, no impact, or a negative impact on each of the following:

		Positive Impact	No Impact	Negative Impact	Not Applicable	Not Asked
A. Goals and Objectives
1.	Pursue innovative ideas.............................................................	87%	2%	<1%	<1%	10%
2.	Pursue high-risk ideas..............................................................	76%	9%	<1%	4%	10%
3.	Obtain other funding.................................................................	54%	26%	5%	3%	10%
B. Applications and Outcomes
4.	Disseminate research findings...................................................	74%	14%	<1%	1%	10%
5.	Develop instrumentation or other enhancements for the research and education infrastructure......................................................	61%	16%	<1%	13%	10%
6.	Develop partnerships with industry, other educational institutions, or national laboratories..............................................................	62%	19%	<1%	8%	10%
7.	Integrate research activity into your teaching and training.............	73%	13%	<1%	2%	10%
8.	Nurture connections between research activity and its potential for: health benefits, economic benefits, and national security benefits...................................................................................	48%	24%	<1%	16%	10%
9.	Develop programs with K‑12 teachers and/or students.................	32%	36%	<1%	20%	10%
10.	Improve public understanding of the project.................................	58%	25%	<1%	5%	10%
C. Process and Team Building
11.	Collaborate with researchers in your area of research..................	83%	6%	<1%	<1%	10%
12.	Broaden participation of under-represented groups in the research activity....................................................................................	62%	23%	<1%	3%	10%
13.	Collaborate with researchers in different areas of research............	76%	12%	<1%	2%	10%
14.	Achieve the research objectives within the specified time.............	83%	6%	<1%	1%	10%
15.	Obtain quality personnel...........................................................	76%	9%	<1%	3%	10%
16.	Establish mentoring or other research-based education activities..	71%	14%	<1%	3%	10%
D. Research Tools
17.	Access state-of-the-art equipment.............................................	60%	22%	<1%	7%	10%
18.	Access facilities.......................................................................	49%	32%	<1%	8%	10%

SKIP Q3.7a IF NO POSITIVE ITEMS IN Q3.6

3.7a Among the items you marked “Positive Impact,” please rank order (write in the number(s)), up to three, those that had the most positive impact.

SKIP Q3.7b IF NO NEGATIVE ITEMS IN Q3.6

3.7b Among the items you marked “Negative Impact,” please rank order (write in the number(s)), up to three, those that had the most negative impact.

Original PI grants in NSF data file	6,180
PIs with multiple grants randomly selected a single grant for the survey (375) or questionable grant information (12)	5,793
Total completes and partials	5,221
Cases screened out during quality assurance process for criteria such as inconsistent grant award or duration information	232
Total cases used for analysis	4,989

January 2001	NSF Director Dr. Rita R. Colwell sends institution presidents an e-mail message announcing the two surveys.
January 24, 2002	MPR sends e-mail messages to institution contact people identified on the NSF data file to identify the appropriate person to participate in the survey.
February 15-March 6, 2002	Questionnaire mail packets are sent as institutional representatives contact information is identified.
March 8-30, 2002	MPR contacts non-responders in the institution sample by phone and e-mail.
March 30, 2002	All data collection is completed.

Total institutions with 2001 NSF grant recipients	582
No contact information	60
Total number with contact information	471(total); 105 (sample)
Total questionnaires returned	369 (total); 95 (sample)
Questionnaires not acceptable after quality assurance	359 (total); 95 (sample)

C. SAMPLE APPROACH

1. Principal Investigator Survey

2. Institutional Survey

January 2001	NSF Director Dr. Rita R. Colwell sends PIs e-mail message announcing the survey.
January 30, 2002	MPR begins sending PI e-mail invitations with Web site access username and password on a rolling schedule.
February 4-19, 2002	MPR sends e-mail remiders to non-responders on a 3 day schedule.
February 15, 2002	MPR sends questionnaire mail packets to 778 PIs who have responded to the Web questionnaire.
March 8, 2002	Deadline for data collection