|
ELT News, February 2010
The KPG Speaking Test Defining
Characteristics
By Dr Kia Karavas
It is
generally acknowledged that the
assessment of speaking is one of the
most challenging and complex areas
of assessment. This is due to
various reasons including: a) the
difficulty of developing an
operational construct definition of
oral performance that can capture
the richness of oral communication
and human interaction; b) the
multiplicity of communication skills
required for oral interaction –skills
that do not lend themselves easily
to objective assessment; c) the
range of factors that influence our
impression of how well someone can
speak a language (cf. Fulcher 2003,
Kitao and Kitao 1996, Luoma 2004).
Of course, despite the difficulties
involved in assessment, speaking is
one of the most common human
activities, the development of the
ability to perform orally an
important part of foreign language
programmes and, therefore, important
in language testing. The KPG exam
battery considers the assessment of
speaking has equal weight as all the
other areas of communication –reading
and listening comprehension, as well
as writing performance.
Speaking is assessed in Module 4 (oral
production and mediation) of the KPG
exams, at all levels. The
development of the speaking test is
based on clearly laid out test
specifications (which all languages
certified through the KPG
examination system follow). For
the design of test items and tasks,
the KPG candidates’ age, first
language, language learning
background and other sociocultural
factors are taken into account. The
test items and tasks undergo
rigorous and systematic pre- and
pilot testing and evaluation (by
oral examiners, KPG candidates and
trained judges) on the basis of
which changes, improvements and fine
tuning of the test items are made.
Moreover, research is carried out by
the RCeL, focusing
on the construct validity and
effectiveness of the test as a whole.
The KPG exams adhere to a functional
approach to language use and set out,
throughout all modules, to evaluate
socially purposeful language use,
which entails a certain amount of
social and school literacy. Within
the KPG exam battery, language is
viewed as social practice embedded
in the sociocultural context by
which it is produced. This view of
language is reflected in the design
of the speaking tasks and in the
assessment criteria.
The tasks rubrics always make the
context of situation explicit, since
candidates are expected to produce
socially meaningful language given
the social context. The sample task
below (from the B2 level exam of May
2007) illustrates this.
|
Speaking Task
Look at this photo. Imagine
you have taken your 4 year-old
niece for a walk in the
centre of town and you come
across this event. Explain
what’s happening, and why
the event is taking place. |
As one
can see, the rubrics for the task
above define, (a) the purpose of
communication (to explain what’s
happening), (b) the content of
communication (to describe the event),
(c) who the participants in the
communicative event are and what
their relationship is (adult-child,
uncle/aunt –niece), (d) what the
setting of the communicative event
is (in the city centre), and (e)
what the channel of communication is
(face to face interaction).
Given the expectations in
performance, one of the assessment
criteria for all levels is
“sociolinguistic competence,” which
simply means that the candidate is
expected to make lexical and
grammatical choices which are
appropriate for the situation as
determined by the task.
The KPG Speaking test
The table below presents the content
and structure of the speaking test
for the exams now administered by
the KPG. As the reader can see, they
share similarities. However, they
differ decisively in terms of task
difficulty, linguistic complexity of
the expected output and linguistic
complexity of the source texts.
A1+A2 LEVEL SPEAKING TEST |
Target population |
Young candidates aged 10-15
years |
Duration of the test |
20 minutes |
Pattern of participation |
Candidates are tested in
pairs but do not converse
with each other. |
Test content |
Dialogue (5
minutes for both candidates
)
This is a “getting to know
you” task, which requires
interaction between Examiner
and candidate. Each
candidate is asked four (4)
questions – two for A1 and
two for A2 level – which are
signposted for the examiner. |
Talking about photos (5
minutes for both candidates)
This activity essentially
involves a guided
description of a photo or
series of photos (or other
visuals e.g. sketches,
drawings) which are
thematically linked. The
activity comprises four (4)
questions – two for A1 and
two for A2 level. |
Giving and asking for
information (5
minutes for both candidates)
This activity is based on
multimodal texts and also
consists of two parts and
five (5) questions in total
–two for A1 and three for A2
level. |
B1+B2 LEVEL SPEAKING TEST |
Target population |
Candidates aged 15+ |
Duration of the test |
15-20 minutes |
Pattern of participation |
Candidates are tested in
pairs but do not converse
with each other. |
Test content |
Dialogue (3-4
minutes) for both candidates
between examiner and each
candidate who answers
questions about him/herself
and his/her environment
posed by the examiner. |
One-sided talk (5-6
minutes for both candidates)
by each candidate who
develops a topic on the
basis of a visual prompt. |
Mediation (6
minutes for both candidates)
by each candidate who
develops a topic based on
input from a Greek text. |
C1 LEVEL SPEAKING TEST |
Target population |
Candidates aged 15+ |
Duration of the test |
20 minutes |
Pattern of participation |
Candidates are tested in
pairs but do not converse
with each other. |
Test content |
Warm-up (not
assessed – 1 minute per
candidate) Examiner asks
each candidate a few ice-breaking
questions (age, studies/work,
hobbies) |
Open-ended response (4
minutes for both candidates):
The candidate responds to a
single question posed by the
examiner expressing and
justifying his/her opinion
about a particular issue/topic. |
Mediation and open-ended
conversation (15
minutes for both candidates):
Candidates carry out a
conversation in order to
complete a task using input
from a Greek text. |
As can
be seen above, the speaking test for
the A1+A2 level integrated test, as
well as for the B1 and the B2 level
tests consist of three activities.
The first one involves questions
relating to candidates’ immediate
environment, work, hobbies, interest
etc., while the C1 level speaking
test consists of two activities the
first of which requires candidates
to respond to an opinion question.
The A1+A2 level speaking test does
not include mediation activities
which are included in the B1, B2 and
C1 level tests. That is, candidates
at this level are required to
perform orally in English, relaying
information they have selected from
a source text in Greek, so as to
respond to a given communicative
purpose.
The speaking test procedure
The KPG speaking test involves two
examiners and two candidates in the
examination room. One of the two
examiners is the ‘Interlocutor’, i.e.,
the one who conducts the exam (in
other words, asks the questions,
assigns the tasks and participates
in the speech event). The other is
the ‘Rater’, i.e., the one who sits
aside silent and evaluates and marks
the candidates’ performance. The
Interlocutor also marks their
performance, once candidates have
left the room. Examiners alternate
in their role as Interlocutor and
Rater every three or four testing
sessions.
Examiners are trained for their
roles as Interlocutor and Rater
through seminars which take place
systematically throughout the year.
However, on the day of the oral
test, examiners are given an
examiner pack which contains
guidelines regarding the exam
procedure and oral examiner conduct,
and the oral test material:
questions, tasks and rating
criteria. These are handed to
examiners at least two hours before
the exam begins along with the
Candidate Booklet which contains the
prompts for the exam (photos and/or
Greek texts, since one of the
activities involves mediation).
Assessment criteria for oral
production
The assessment criteria for oral
production reflect the functional
approach to language use that the
KPG exams adhere to. The structure
of the assessment scales for the
different level speaking tests are
similar, while the nature of the
criteria depends on the expectations
for oral production, illustrative
descriptors and can-do statements
for each level and on the
requirements of the speaking
activities. Thus for the A1+A2, the
B1 and the B2 level speaking tests,
the assessment criteria are grouped
under two main categories: criteria
for assessing a) task completion and
b) language performance. As a
result, the Speaking Test Rating
Scale, which provides a breakdown
and description of each criterion on
the oral assessment scale, is
presented in two parts. Part 1
presents the Task
completion criteria,
which have to do with the degree to
which the candidates achieved the
communicative purpose of the task,
while Part 2 presents the Language
performance criteria,
which focus on the quality of
language output in terms of
pronunciation, vocabulary, grammar
and syntax, and coherence (for
A1+A2), and in terms of the
candidates’ phonological competence,
linguistic competence,
sociolinguistic competence and
pragmatic competence for the B1 and
B2 level speaking tests.
Given the requirements of the C1
speaking test, the assessment
criteria are grouped under two
categories: a) overall performance
for tasks 1 & 2, and b) assessment
of task 2: the mediation activity.
As a result, the Speaking Test
Rating Scale, which provides a breakdown
and description of each criterion on
the oral assessment scale, is
presented in two parts. Part 1
presents the Overall performance for
tasks 1 & 2 criteria which focus on
the quality of language output in
terms of the candidates’
phonological competence, linguistic
competence, appropriateness of
language choices, and cohesion,
coherence of speech and fluency;
Part 2 presents the Assessment of
task 2 criteria, which focus on the
interaction and mediation skills of
the candidates.
Developments and research
relating to the speaking test
Assessment of oral proficiency is a
complex and largely subjective
process in which many variables
affect the quality and quantity of
language output, while the rating of
performance ultimately threatens the
validity, reliability and fairness
of the oral test procedure. Given
that one of the most significant
variables potentially affecting
candidate output and examiner rating
is the linguistic conduct of the
examiner (Bachman et al. 1995; Bonk
& Ockey 2003; Lazaraton 1996,
McNamara 1996; O’Sullivan 2000) ,
the KPG exam system has established
a systematic, intensive and on-
going programme of oral examiner
training. The
goal of the training programme is to
develop a database of approved oral
examiners; that is examiners who
have taken part in all training
seminars and whose performance has
been positively evaluated by
specially trained observers. The
Observation Project, which has
yielded interesting data now
systematically being analyzed, will
be hosted in the KPG Corner of the
next issue of ELT
News.
Another aspect of the speaking test
being investigated systematically is
the reliability of examiner marking.
This is monitored and assessed
through systematic inter-rater
reliability checks during/after each
exam administration. For the
speaking test, inter-rater
reliability is assessed through the
data collected by observers who note
the mark for each candidate awarded
by each observed examiner. On the
observation form, the observers also
note their mark of candidates’
performance. This data is then
collected by the RCeL, where inter-rater
reliability estimates are calculated
and results are interpreted.
Finally, after each exam
administration, oral examiners are
requested to complete an oral test
feedback form which aims to elicit
feedback concerning the potential
problems with test items, their
usefulness, appropriateness and
practicality. These results are sent
to the project team directors, who
share and discuss them with their
test development teams. The project
team undertakes a systematic
post-examination review, weighs the
results of the exams, and takes into
consideration the responses on
examiner feedback forms before
deciding on potential
revisions/changes deemed necessary
for the improvement of the test
papers.
References
Bachman, L. F., Lynch, B. K., &
Mason, M. 1995. Investigating
variability in tasks and rater
judgements in a performance test of
foreign language speaking. Language
Testing 12,
239-258.
Bonk, W.J. & Ockey, G. 2003. A many-facet
Rasch analysis of the second
language group oral discussion task. Language
Testing 20
(1), 89-110.
Fulcher, G. 2003. Testing
Second Language Speaking.
Harlow: Pearson Education Limited.
Karavas, K. (ed.) 2008 The
KPG Speaking Test in English: A
Handbook. National
and Kapodistrian University of
Athens, Faculty of English Studies:
RCeL Publication Series 2 (RCeL
publication series editors: Bessie
Dendrinos & Kia Karavas).
Kitao, S.K. & Kitao, K. (1996).
Testing speaking. ERIC Clearinghouse
ED39821.
Lazaraton, A. 1996. A qualitative
approach to monitoring examiner
conduct in CASE. In M. Milanovic and
N. Saville (eds.) Performance
Testing, Cognition and Assessment,
Studies in Language Testing 3, Selected
Papers from the 15th Language
Testing Research Colloquium,
Cambridge and Arnheim, Cambridge:
UCLES/Cambridge University Press,
18-33.
Luoma, S. 2004. Assessing
Speaking. Cambridge: Cambridge
University Press
MacNamara, T. F. 1996. Measuring
Second Language Performance.
London: Longman.
O’Sullivan, B. 2000. Exploring
gender and oral proficiency
interview performance. System 28,
373-386.
<Πίσω> |
|
|