NAEP Technical DocumentationProcessing of Student Booklets and Questionnaires
| |||
There are several stages of work involved in receiving and processing the documents used in NAEP paper-based assessment (PBA) and digitally based assessment (DBA).
Starting in 2017, the DBA response data and questionnaires (received through the online questionnaire system) have been securely transferred to the materials, distribution, processing, and scoring (MDPS) staff. The MDPS technology team processes the data, validates the responses are provided as expected, and converts the responses into a more easily consumable format for the design, analysis, and reporting (DAR) staff. Questionnaire data is formatted by the NAEP platform development (NPD) staff. Digital responses are passed along to DAR staff as is; any response submitted on paper forms are checked against digital responses to remove duplicates and formatted to match the digital response files.
The MDPS staff create a set of predetermined rules and specifications for its processing departments to follow the PBA session materials. A variety of procedures are performed on materials received from the sampling and data collection (SDC) administrators before releasing these materials into their processing system. Control systems are used to monitor all NAEP materials returned from the field. MDPS internal databases contain the status of sampled schools for all sessions and their scheduled assessment dates. As materials are returned, the databases are updated with receipt dates and documents any problems discovered in the shipments. As documents are processed, the databases are updated to reflect processed counts. The materials report programs are utilized to allow MDPS staff to monitor the progress of the receipt control operations.
An alert process is used to record, monitor, and categorize all discrepant or problematic situations. Throughout the processing cycle, alert situations are either flagged by computer programs or identified during clerical check-in procedures.
Certain alerts, such as the administration code on the booklet cover does not match what is written on the
administration form, can be resolved by MDPS staff accessing the information within the SDC's electronic
preassessment visit system. Since these problem situations are categorized and tallied as they are entered in the databases, project staff are able to provide timely reporting to the SDC staff these clerical-type errors made during test administration.
In cases where an alert situation cannot be resolved by the MDPS opening staff, an alert form is completed and forwarded to project personnel for resolution. An example of this type of alert would be a session returned with no administration form, so staff are unable to check in the booklets.
The
Work Flow Management System (WFM) created for NAEP is used to track batches of student booklets through each processing step, allowing the MDPS staff to monitor the status of all work in progress. WFM is also used to analyze the current workload, by project, across all workstations. Through consistent monitoring of these data, the MDPS staff are able to assign priorities to various components of the work and to monitor all phases of the data receipt and processing.
After the PBA student booklets and questionnaires are processed, they are stored securely in the MDPS warehouse until NCES provides approval to securely destroy them. The electronic data (multiple-choice, open-ended responses, and human scores) captured from PBA and DBA are stored on a secure server until given approval by NCES to be securely destroyed.
Last updated 18 July 2024 (SK)