White Papers and Research Summaries
Passport Language Program Validation Study
Download the pdf!Passport Language Program (PLP) consists of two exams (entrance and certification) to measure English proficiency for students in high school and beyond. This document contains a psychometric analysis conducted on test data from the Listening and Reading portions of both the PLP B1 Entrance and Certification tests. (2008)
Exploring Alternative Methods of Evaluating Written Communication Skills in the Admission Process
Download the pdf!A prospective, cross-sectional, comparative study investigating alternative methods that may be used to evaluate candidates' written communication skill was conducted. (2008)
Journal of Technology, Learning and Assessment, March 2006
Download the pdf!A 2006 study published in the Journal of Technology, Learning and Assessment, a free online journal published by Boston College, states the "IntelliMetric® system is a consistent, reliable system for scoring AWA essays." (The AWA is the writing portion of the GMAT® assessment.) The study analyzes IntelliMetric® by first comparing its scores to individual human raters using "a Bayesian system employing simple word counts, and a weighted probability model using more than 750 responses to each of six prompts." The second evaluation was larger and compared "the IntelliMetric® system ratings to those of human raters using approximately 500 responses to each of 101 prompts."
The study found IntelliMetric® had "a perfect + adjacent agreement on 96% to 98% and 92% to 100% of instances in evaluations 1 and 2 respectively" and that "correlations of agreement between human raters and the IntelliMetric® system averaged .83 in both evaluations."
The Effects of Inclusion of Native Speakers' Writing Samples on the Domain Scoring Accuracy of Automated Essay Scoring of Writing Submitted by Taiwanese English Language Learners
Download the pdf! Login required.Presented at the 32nd Annual Conference of the International Association for Educational Assessment in Singapore in May 2006, this paper presents findings regarding the effects of training set composition on the domain scoring accuracy of essays submitted by Taiwanese students scored by an automated essay scoring system. This study compares the accuracy of scoring the same set of essays written by Taiwanese students using two different models: one model using blended native and ELL essays and one using a set of entirely ELL essays. Read more.
IntelliMetric® Accuracy Summary Report
Download the pdf!Presented at the Annual Meeting of the American Educational Research Association in San Francisco, California in April 2006, this paper evaluates the scoring accuracy of models created under the conditions of prompt-specific and prompt-independent training in order to investigate the efficacy of these two approaches.
Necessary Components for Effective Writing Instruction
Download the pdf!Explores successful instructional strategies that can be used to improve the writing skills of students and to increase the amount and frequency of writing.
Effects of Mode of Delivery for Constructed Response and Selected Response Assessments
Download the pdf!Examines whether the delivery mode of an assessment, specifically traditional paper and pencil versus computer-based, impacts on the results of the test. The two delivery modes are compared for both multiple-choice and constructed-response assessments as they relate to those who are taking the exams and who are judging (grading) the exams.
WritePlacer Validity Study
Download the pdf!Presented at the annual meeting of the American Educational Research Association, Division D, in Montreal, Canada in 2005, this paper reports the results of a three-year study of a direct writing assessment program incorporating automated essay scoring technology used for college placement. The study, conducted at more than 100 colleges and universities throughout the United States, examines the validity of the instrument using a multi-trait multi-method approach. Several hundred thousand students took the direct writing assessment and one or more other measures as a basis to examine the construct validity of the writing component.
This study examines the validity of the test scores for WritePlacer®, a direct assessment of writing included as one component of The College Board's ACCUPLACER® college placement testing program, and investigates the extent to which WritePlacer® scores relate to other measures of writing and other academic skill areas.
A Comparison of the Accuracy of Bayesian and IntelliMetric® Automated Essay Scoring Methods to Score Essays Written in English
Download the pdf!Presented at the 10th Annual National Roundtable Conference in Melbourne, Australia in 2005, this paper investigates the scoring quality of IntelliMetric® compared to Bayesian automated scoring methods. IntelliMetric® has been shown to be an effective tool for scoring essay-type, constructed response questions across K-12, higher education and professional training environments as well as within a variety of content areas and assessment purposes.
Please contact us to learn more.