In this
issue the KPG corner is concerned
with the evaluation criteria for the
writing test paper of the English
exam, i.e., Module 2. The philosophy
of this test paper is based on a
functional view of language and a
genre approach to writing assessment.
Genre-based approaches emphasize the
social constructedness of language
and acknowledge that, while language
is produced by individuals, its
shape and structure are to a great
extent socially determined. Taking
into account that in everyday life
we write as members of specific
communities, producing texts which
conform to different social rules (influenced
by a variety of contextual factors,
such as who is writing what to whom
and for what purpose), similarly in
assessment conditions, we ask our
candidates to produce scripts which
take into consideration specific
contextual factors. In fact, the
rubrics (instructions) of the
writing activities require
candidates to assume a role in order
to produce a text of a particular
text type, addressing a specific
audience and meeting a predefined
communicative purpose; that is, they
determine to a great extent the
content, text organization and the
language to be used in a candidate's
script.Consequently, when evaluating
scripts, KPG raters are trained to
assess candidates' ability to
produce language which is
appropriate for the situational
context rather than simply correct
in terms of form.A second rule of
the KPG writing evaluation system is
to have achievement as the starting
point rather than failure. In other
words, raters are trained to
consider what candidates have
managed to accomplish rather than
what they have failed to do. In fact
the KPG rater guide and the rating
grid serve as tools to help raters
focus on candidates' communicative
performance rather than the
grammatical, lexical, spelling or
punctuation errors they might have
made when writing. In assessing
performance, they are guided to
consider what candidates have done,
how well they have responded to the
writing activity and the degree to
which they have used structures and
forms appropriately, given the
context which is always specified.
Evaluation criteria
There
are three evaluation criteria which
aim at helping raters focus not
merely on sentence grammar and
lexis, but on discourse and text as
well as sentence grammar.
Evaluation criterion
1
The
first evaluation criterion has to do
with task
completion and
is directly related to contextual
features, i.e. the communicative
purpose of
the produced script, its
appropriateness in terms of genre, register and style.The
notion of genre is
used here to refer to both text
type and generic
process. A genre as text
type (e.g.
new report, email, recipe, film
review) is characterised by
relatively stable structural forms
(e.g. particular beginnings, middle
and ends), particular ways of
organizing information (e.g. in
paragraphs or in bullet forms) and
lexicogrammatical features and
patterns used according to the
social purpose of the text. However,
texts are not only determined by an
overall social purpose but they are
also formed out of the dynamics of
social processes,
such as to instruct, to argue, to
narrate, to describe or to explain.
Each one of these processes is
associated with different language
features; for example, a newspaper
article whose purpose is to report a
racist event employs different
language features from an article
whose purpose is to argue against
racism. Raters are trained to assess
the degree to which a script has
addressed the task set and has
developed it in terms of theme/topic. They
also consider whether candidates
have produced an appropriatetext
type and
responded to the required generic
processes.For example, let us
say that the writing activity
requires candidates to produce a
report (text type) which
informs readers (generic
processes) about the work of a
volunteer programme (theme)
and that instead of producing a
report some candidates produced a
letter. This would be partially ok,
if these candidates managed at least
to inform readers about the
volunteer programme's work. However,
if instead of informing about the
work that the programme involves
candidates provide information about
the advantages of joining the
programme, they may have failed to
meet Criterion 1, especially because
it is likely that they have not used
appropriate register and style,
which are additional requirements of
this criterion -depending of course
of the level of language proficiency
being tested.
Evaluation criterion
2
Criterion 2 is related to text
grammar (text
organization, coherence and
cohesion). The notion of text
grammar as
understood here addresses issues
above the sentence level. Raters are
trained to assess the degree to
which candidates have managed to
produce a coherent and cohesive
script. Coherence refers
to the presentation of ideas in a
logical and understandable way.
Candidates are expected to produce
coherent texts by drawing on
knowledge of how to organize and
present their ideas from their
previous experience as text
producers and from their experience
as readers. For instance, they know
that events in a story are presented
in chronological order, while
arguments in an essay are often
presented in terms of their
importance (starting from the less
important and moving to the most
important arguments, or the
opposite).Candidates are also
expected to produce cohesive texts. Cohesion refers
to the ways a part of a text is
linked to another part of the text
and it can be achieved in a script
through a variety of ways: through
the use of connectives (linking
words and expressions), pronoun
reference, repetition of key words,
etc:
Connectives indicate
how an idea presented in one
sentence relates to the next one
(e.g. in an antithetical way through
the use of connectives such as but, on
the other hand, however,
etc). The use of connectives is an
indicator of writing development:
the more advanced a candidate's
level of language competence, the
more complex and logical connectives
that s/he uses for the construction
of complex sentences. However, some
candidates make repeated use of some
formal connectives (e.g. in
addition, furthermore, to conclude)
which are to be found in a formal
essay, but which may be
inappropriately used in various
other genres, as well.
Reference is
another way through which cohesion
may be achieved. The use of personal
pronouns is the most common way of
maintaining reference which avoids
the repetition of names. Raters are
trained to take into account the
fact that control of reference is an
indicator of how well the flow of
information from one sentence to the
next or from one part of a script to
another is maintained.
Tense consistency refers
to appropriate use and control of
tenses in a script. The use of tense
changes from genre to genre. For
example, factual descriptions are
generally written in the present
tense, while narratives in the past
tense. Raters are trained to assess
a script's use and control of tense
and possible changes in the same
script.Overall, the criterion of text
grammar considers
how all parts of a script are
structured, organized and coded, in
order to make it effective for the
purposes of a particular
communicative context.
Evaluation Criterion
3
Criterion 3 is related to sentence
grammar and lexical
features. Sentence
grammar refers
to the use of language or
lexicogrammar according to formal
rules of grammar, syntax and
morphology. Raters are trained to
assess candidates' writing
performance considering the features
of their sentences; e.g.,
correctness of clause pattern,
subject-verb agreement, verb form,
preposition, articles, plurals, etc.
Errors are expected in varying
degrees, depending on the level.
However, raters are trained not to
seriously penalize errors as long as
they do not interfere with
intelligibility. The importance of
errors which violate rules of formal
grammar are assessed on the basis of
whether a script manages to convey a
socially meaningful message, despite
formal errors.Concerning lexical
features, different
text types require use of different
types of vocabulary as well as a
different range of vocabulary (or
repetition), depending on
determining categories such as topic,
purpose and audience. For example,
an academic report will use a range
of technical vocabulary including
nominalizations and technical noun
groups; a literary description, on
the other hand, will use descriptive
verbs, adjectives and adverbs, and
affective language in order to
create an emotive effect on the
reader. Raters should assess
vocabulary appropriacy in terms of a
specific text type and lexical range
(for more advanced levels). Spelling
errors are considered differently,
depending on whether or not they
interfere with intelligibility and
depending on the level.
Punctuation and
writing conventions are also
assessed when considering Criterion
3.
Applying the
evaluation criteria
The rating process is
done on the basis of a rating grid
which guides raters to follow a
procedure which moves from a
holistic view (i.e. overall
impression) to finer points of
assessment. Though the grid for each
level is different, all grids have
been designed on the basis of the
same philosophy and therefore, they
can be applied by using the same
methodology. The idea behind it is
to have a zone based assessment
rather than to subtract points on
the basis of how many errors a
script has. The decision the rater
has to make, by moving from right to
left on the grid and from left to
right, is to decide whether the
script is fully
satisfactory(responding to all
three evaluation criteria), moderately
satisfactory (satisfying
some of the criteria or satisfying
them partially) or unsatisfactory (partly
responding to a limited number of
criteria or points of the criteria).
The application of
the rating grid is generally a
demanding process. In order to
ensure reliable assessment and
marking, the system needs to make
sure that during the script rating
process, raters use the rating grid
systematically and correctly. In
order to achieve this goal the
system takes a variety of measures.
The two most important ones are:
-
To produce
material that provides very
concrete guidelines as to how to
assess and mark KPG writing
scripts. It is for this reason
that for every exam period a
very detailed Rater Guide is
produced and given (free of
charge) to KPG raters. This
includes analyses of the
evaluation criteria, the rating
grids and guidelines regarding
how to mark the scripts of the
particular exam period with
articulated expected outcomes
and real sample scripts which
have already been marked. To
help further, a Handbook for
this purpose is being prepared.
It is an RCeL publication, to
appear within the RCeL
publication series, of which the
editors are Bessie Dendrinos &
Kia Karavas. The Handbook (edited
by myself) is entitled The
KPG Writing Test in English.
-
To
systematically work with raters.
In the last issue of the ELT
News, the "KPG Corner"
presented the KPG script rating
programme which runs every exam
period before script assessment
begins. The overall aim of this
programme is to train raters to
be able to reliably mark the
scripts which have resulted from
the writing activities of that
period's exams. In order to
enhance assessment reliability,
KPG scripts are evaluated and
marked by two raters,
and the final mark of each
script is the average of their
marks. If there is high
discrepancy between them,
scripts are re-evaluated, and
the incident a cause for concern.
The raters involved are informed.
Of course, to avoid such
incidents, raters are
consistently supervised by
highly qualified coordinators
who are also assigned with the
responsibility of evaluating the
raters for intra- and inter-rater
reliability.
Bessie Mitsikopoulou
Assistant Professor, Faculty of
English Studies
