Computerized Adaptive Testing - How CAT Works

How CAT Works

CAT successively selects questions so as to maximize the precision of the exam based on what is known about the examinee from previous questions. From the examinee's perspective, the difficulty of the exam seems to tailor itself to his or her level of ability. For example, if an examinee performs well on an item of intermediate difficulty, he will then be presented with a more difficult question. Or, if he performed poorly, he would be presented with a simpler question. Compared to static multiple choice tests that nearly everyone has experienced, with a fixed set of items administered to all examinees, computer-adaptive tests require fewer test items to arrive at equally accurate scores. (Of course, there is nothing about the CAT methodology that requires the items to be multiple-choice; but just as most exams are multiple-choice, most CAT exams also use this format.)

The basic computer-adaptive testing method is an iterative algorithm with the following steps:

1. The pool of available items is searched for the optimal item, based on the current estimate of the examinee's ability
2. The chosen item is presented to the examinee, who then answers it correctly or incorrectly
3. The ability estimate is updated, based upon all prior answers
4. Steps 1–3 are repeated until a termination criterion is met

Nothing is known about the examinee prior to the administration of the first item, so the algorithm is generally started by selecting an item of medium, or medium-easy, difficulty as the first item.

As a result of adaptive administration, different examinees receive quite different tests. The psychometric technology that allows equitable scores to be computed across different sets of items is item response theory (IRT). IRT is also the preferred methodology for selecting optimal items which are typically selected on the basis of information rather than difficulty, per se.

In the USA, the Graduate Management Admission Test are currently primarily administered as a computer-adaptive test. A list of active CAT programs is found at International Association for Computerized Adaptive Testing, along with a list of current CAT research programs and a near-inclusive bibliography of all published CAT research.

A related methodology called multistage testing (MST) or CAST is used in the Uniform Certified Public Accountant Examination. MST avoids or reduces some of the disadvantages of CAT as described below. See the 2006 special issue of Applied Measurement in Education for more information on MST.