Kia Karavas
TRAINING THE SCRIPT RATER

Within the context of the KPG exams, the systematic training of oral and script raters is regarded as one of the most significant factors contributing to the validity and reliability of the writing and the speaking tests (Modules 2 and 4 respectively) and to the sustainability of the examination system as a whole. This text focuses on the main principles and characteristics of the training programme for our body of script raters. 

Who are our script raters?

Script raters are experienced English language teachers who have been chosen from an initial pool of applicants to the Ministry of Education after a successful initial screening interview. Most script raters have been KPG oral examiners as well. As new levels of language competence were introduced in the KPG exam battery, the need for a larger number of script raters arose. Thus, in subsequent phases of the programme, after screening more recent applications from potential script raters, ELT professionals with postgraduate studies in applied linguistics and with experience in marking scripts in other language examinations were invited to become part of the KPG script rater programme. Unavoidably, the area of residence has become a criterion for selection since the KPG marking centre for the English exam is located in Athens and thus only teachers living and working within the Attiki area are able to join the programme and become practicing script raters.

What are the aims of the script rater training programme?

The aims of the English script rater training programme include the development of:

1. a body of 300 script raters who have been fully trained for assessing all levels of the writing module offered by the KPG exam battery and whose performance has been evaluated on the job,

2. a comprehensive and fully updated database of trained script raters which the Ministry of Education can draw from to make appointments for every exam period,

3. comprehensive training handbooks for script raters accompanied by samples of candidates’ written production for self training and awareness raising purposes.

How are script raters trained?

Training before the marking process begins

The training of script raters started from the first exam administration and it gradually became more systematic and principled from the second exam administration onwards. Today, the script rater training programme consists of a series of stages which all script raters are required to go through.

After screening the applications, groups of ‘trainee’ script raters are formed and are invited to a four-hour induction seminar. During this seminar, trainees are provided with a detailed script rater information pack and are informed about the theory of language underlying the writing test, the content and structure of the test for each level, the expectations for written language production for each level and the assessment criteria. The information pack also contains samples of candidate scripts from past examination periods which the trainee script raters evaluate applying the assessment criteria.  New script raters are then invited to the script rater training seminars which take place after each exam administration. They are requested to participate in all the tasks assigned during the seminar. New script raters are required to take part in at least two script rater seminars (apart from the induction seminar) before they are allowed to take part in the marking process.

The script rater seminar takes place two weeks after the actual exams. The preparation for this seminar entails quite a complicated and time consuming process in which many individuals and parties are involved. The preparatory stages for the seminar are as follows:

1. Immediately after the examination, members of the test development team prepare the expected outcomes of each writing task for every exam level in terms of (a) the genre, communicative purpose, register/style, (b) expectations regarding coherence and cohesion and (c) lexicogrammatical choices.

2. At the same time, the committees at the Examination Centres around the country pack the English scripts and send them to the Rating Centre in Athens. There, the Rating Centre committee randomly selects 100 scripts from each level writing task; the selected scripts have been produced by candidates having taken the exam in cities and towns in different parts of the country.

3. After gathering 100 scripts, 20 experts from the KPG English team, who are well aware of what the writing tasks are aiming to test and what the expected outcomes of each writing task are, as these have already been recorded, meet at the Rating Centre and, divided into groups according to test level, use the rating scale and evaluate 100 scripts. Each script is evaluated and rated by two “expert” raters, as when the regular rating process begins.

4. Detailed discussion regarding candidates’ performance on the particular tasks follows with the purpose of (a) assessing task validity, (b) finalizing expected outcomes, (c) fine-tuning the rating scale, (d) selecting the scripts which are best examples of satisfactory and non satisfactory scripts, (e) conferring about scripts that may have resulted in rating discrepancy between experts.

5. On the basis of this discussion, members of the KPG test development team, update, refine and revise the Script Rater Guide which includes information on the nature, structure and main features of the written test for all levels, the criteria for the assessment of candidate’s written production, the writing tasks of each recently administered exam together with detailed information on the expected output for each activity. Sample candidate scripts are included with the marks assigned by the test development team and brief commentaries justifying the assigned mark. Finally, sample candidate scripts are also provided without marks and commentaries to be evaluated by script raters during the seminar.

6. After the production of the script rater guide, our body of script raters (experienced and new) are invited to a one day seminar/workshop. They are provided with a copy of the script rater guide (which they are requested to bring with them at the marking centre and consult when marking proper begins). During the seminar, script raters are informed of the writing tasks of each level and the expected language outcomes, they are presented with candidate scripts and how they were marked by the test development team. The seminar then takes the form of a workshop where script raters are provided with samples of candidates’ answers and are asked to mark them and to justify their mark in relation to the criteria for assessment of written production. Problems or queries with the marking of scripts are discussed and clarified.

Individualised Training at the Marking Centre

The training of script raters continues at the marking centre on an individual basis. Centre coordinators closely monitor the marking process and offer on the spot advice, help and training to script raters. More specifically, our body of script rater coordinators consists of highly qualified and experienced associates who in their majority are members of the KPG development team. Coordinators are present at the marking centre throughout the whole period of the marking process. They work at the centre at predetermined shifts (there are two shifts per day except Sunday when there is only one) taking care that there are at least 2-3 coordinators at the marking centre at any one time. The number of the coordinators needed for each marking period depends on the number of a) candidates of the particular exam period and b) script raters involved in the process in each exam period. The duties of the coordinators are briefly listed below:

1. They advice script raters whenever the latter face a problem with the application of the assessment criteria.

2. They monitor raters’ individual performance during the rating of a certain number (at least three in every packet of 25) scripts in all levels of the exam. This procedure is followed each time raters are obliged to move to the next level of the exam.

3. They monitor the script raters’ application of the assessment criteria in each of these scripts and they keep records of their performance by filling in two different statistical sheets. The first one includes more general comments while the second one is more detailed and asks for the coordinators’ justified evaluation of the individual script raters.

4. Additionally, the coordinators monitor raters’ performance through randomly chosen scripts already marked in which the raters are asked to justify their assigned marks. The coordinators discuss the application of the rating scale and they keep records of the whole procedure. These records are analyzed after the rating period has ended and details of the raters’ actual performance are recorded and analyzed for further reference and evaluation of the individuals and of the process itself.

5. Whenever different coordinators realize that particular script raters need more training on the correct application of the rating scale, they keep working together with these raters until it is decided that they do not need further assistance.

6. At the end of each rating period there follows a statistical analysis of all rating discrepancies between raters which offers additional data for evaluating both the rating process and the performance of the participating raters. Data are also compared with those produced in previous rating periods so the results could be used for the development of a comprehensive and fully updated database of trained script raters. Additionally, coordinators’ and script raters’ feedback forms offer suggestions for the improvement of the rating procedure itself.

Introducing mentor raters for on-site training of new script raters

One of the strategies that were successfully used with new script raters was the Mentorship scheme. A small number of experienced and trained raters were chosen to act as Mentors, that is a body mediating between the coordinators and the raters, whose job would be to help new KPG raters familiarize themselves with the procedure and the assessment scale. This scheme has been successfully used and it is considered a successful way of training both new raters and experienced ones giving them the opportunity to improve acquired rating skills.