Standard Stages in Assessment Instrument Development

Summary #1

Agus Eko Cahyono

Jumariati

================================================================

Standard Stages in Assessment Instrument Development

Developing a good assessment instrument is ideally done through several stages that can guarantee its quality. There are several stages in developing assessment instrument proposed by some experts. Hughes (2003) state that the stages in developing assessment instrument include:

1.      Stating the problem that means specifying what to test and how to test. During this stage, the instrument developer (teacher) needs to be clear with what type of test is going to be developed whether it is an achievement test, a proficiency test, or placement and diagnostic test. Then, he needs to decide the specific goals of the test by deciding the specific areas of knowledge and/or specific skills to measure.

2.      Writing the specification of the test. In this stage, the instrument developer has to determine the specifications of the test regarding the content, structure, numbers of the item, the test medium whether paper-pencil based or computer-based, techniques of measurement, levels of performance, and scoring procedure.

3.       Writing and moderating item. During the stage, the instrument developer deals with deciding certain samples of content to be tested, writing the test items, and moderating the test which can be done by involving other colleagues or experts in checking the test items.

4.      Informal trialing of items with native speakers. This can be done with only twenty people having similar characteristics with the test-takers. The aim of informal trialing is to check whether each test item is appropriate or not, whether they are ambiguous, too difficult, and too easy. If they are so, the items need to be revised before given to the real test takers.

5.      Trialing of the test on a group of non-native speakers similar to those for whom the test is intended. After being moderated and informally trialed, the test items are given to the non-native speakers similar to the real test-takers. The aim is to anticipate any problems during the test administration and the scoring. Sometimes this stage is not feasible due to the difficulty in finding the parallel group of trialing and the security of the test.

6.      Analyzing the results of the trial and making any necessary changes. This is done statistically and qualitatively. The results of the analysis can show the quality of the test in terms of the reliability and discriminating power and also to avoid faulty items.

7.      Calibrating the scales if the rating scale is used. This is done by collecting samples and assigning each item using the relevant scales of the rating.

8.      Validating the test. This is necessary to do especially with high stakes test (published test) and also low stakes test that is going to be used for several times in an institution.

9.      Writing handbooks for test takers, test users, and staffs. The handbooks contain information on the rationale of the test, how it was developed, sample item, advice for taking the test, score interpretation, and test administration.  

10.  Training staff which means preparing others (interviewers, raters, scorers, computer operators, ad proctors) to be involved in the test process.

Meanwhile, Brown and Abeywickrama (2010) propose several stages of assessment instrument development, they are:

1.      Determining the purpose of a test. By doing so, the test designers can determine the usefulness of the test.

2.      Designing clear, unambiguous objectives meaning that the test designers (teachers) need to be explicit in stating the test objectives based on the goals of instruction stated in the curriculum.

3.      Drawing up test specifications which serves as a blueprint of the test. It should contain the information on the description of the test content, item types, tasks, skills to be included, scoring, and reporting.

4.      Devising test items by making the draft of the test items regarding the test type, test content, test objective, and the difficulty level. Inviting colleagues to look at the draft is necessary to estimate its quality. When the draft looks improper, it needs some revision before proceeding to writing the final draft.

Following these standards stages can help test developers to design a good assessment instrument even though some of the stages are almost not feasible due to some factors. However, the stages above can become the guidelines for teachers as test developers to develop a qualified assessment instrument which can provide good feedback on their teaching practices. 

References:

Brown, H.D. & Abeywickrama, P. 2010. Language Assessment: Principles and Classroom

Practices. Second Edition. White Plains: Pearson Education, Inc.

Hughes, A. 2003. Testing for Language Teachers. Cambridge: Cambridge University Press.