SESTAT Survey Design and Methodology

Editing Guidelines and
Procedures

Editing Guidelines and Procedures

The three SESTAT surveys were conducted by three different survey data collection contractors, depending on the year. As a consequence, NSF developed standardized guidelines so that all contractors used the same editing procedures for their respective surveys. Certain data were deemed "critical data elements" that needed to be complete and consistent before a data record was considered complete.

After status determination and data entry, general editing was performed and the "best coding" and "other, specify" coding procedures were completed. The editing rules include (1) valid code range edits, (2) skip error edits, (3) mark one edits for question with more than one response marked, and (4) consistency edits. Procedures were developed for general editing rules such as distinguishing between questions that had "refused," "don't know," or "blank" responses; for rounding rules for decimals or fractions; for missing data on questions with a series of "yes/no" responses; for number of employees; for coding primary and secondary work activities, and most and second-most important reason for working outside field of highest degree; and for most important reason for attending training.

Occupation and Education "Best" Coding

The purpose of "best coding" procedures for occupation and degrees was to correct respondent reporting problems. Typical respondent errors included such things not making a code entry or not reviewing the entire list before making a selection. Recurring problems were found with respondents distinguishing managerial and teaching occupations

Special coding procedures were developed to increase the data quality and comparability for occupation and education codes. In most cases, the respondents self-selected their occupation and education categories from job and education code lists at the end of the questionnaire. The remainder were chosen by CATI respondents through a series of questions that began with the broad categories and narrowed the selection to the specific category.

The special coding procedures focused on correcting respondent reporting errors. However, an important "best coding" rule was that the coder should not change the respondent chosen code unless there was clear evidence that the respondent's choice was incorrect.

During "best coding" the coder reviewed a variety of respondent-provided information and used standardized references and procedures.

The "best code" for occupation was determined by reviewing factors such as

Respondent's open-ended (verbatim) response and self-selected code
Employer name and address
Type of employer: educational institution, business, government, etc.
Business of primary employer
Number of people respondent supervised directly and indirectly
Relationship between respondent's work and highest degree
Field and level of degrees held by respondent
Primary, secondary and other work activities
Salary
For CATI respondents only -- which broad category was chosen first
For SDR respondents only -- tenure status
Marginal or interviewer notes where available
Occupational information from prior years

The "best code" for a degree held by the respondent was determined by using one of two "flow charts." One flow chart outlined the procedures for dealing with verbatim responses that list one major field of study. The other flow chart outlined the coding procedures for a response that gave more than one field of study. These flow charts standardized the coding procedures and gave special procedures for situations involving exact verbatim matches; handling single, broad, or nonspecific field matches; and rules for assigning the most specific NSF education code. "Best codes" for education were assigned after determining whether the respondent selected a code that was too general, transposed the code numbers, or wrote the numbers incorrectly. Education codes were not "best-coded" in three cases: if the respondent-selected code was more specific than the respondent verbatim and both verbatim and code are in the same field; if the verbatim response was more specific than the self-selected code, and both were in the same field; or if the verbatim response and the selected code fell under the same broad educational category. Only when it was evident that the self-code was incorrect was a "best code" assigned.

"Other, Specify" Coding

The purpose of editing "other, specify" responses was to identify responses that belonged in specific existing categories. This procedure is called "back-coding." "Other, specify" responses often fell into one of the following categories: an existing response category; legitimate "other" response; or not a legitimate response (i.e. does not answer the question). Other responses that fell into the first category were back-coded.

Data Collection

Missing Data Imputation