ORAL EXAMINERS
Oral examiners’ attitudes
towards the exam and
specific activities are
investigated through the
use of questionnaires
distributed to examiners
in all exam centers on
the day of the oral
exams. These feedback
forms include questions
about the oral test
procedure, the examiner
conduct, the format and
the content of the oral
test papers and
communication
strategies.
SCRIPT RATERS
Script raters’ views about the rating procedure, the script rating training, the
script rater tools and their marking experience are also explored through the
use of questionnaires completed at the end of the script rating procedure. In
order to assess the quality of script evaluation, the ways in which script
raters use evaluation criteria are systematically observed, inter-relater
reliability is conducted and the characteristics of the scripts which cause
serious problems in inter-relater reliability are thoroughly examined.
CANDIDATES
Through specially
designed questionnaires
distributed in each
examination period, KPG
candidates’ views and
attitudes towards test
papers are also being
explored with a view to
discovering what types
of tasks pose
difficulties to
candidates, what factors
they think affect their
performance, what
strategies they use in
order to perform well or
in what themes they are
particularly interested.
Statistical analysis is
then conducted (through
SPSS or other tools),
the results of which are
exploited for the
development of reports
referring to the main
research results.
THE
ITEM ANALYSIS PROJECT
This
project aims at
determining the degree
of item and task
difficulty of the
reading and listening
comprehension test
components (Modules 1
and 3). Using
examination data the
Ministry of Education
provides us with after
each exam
administration, we
calculate through
specialized software the
following statistical
measures: a) internal
consistency or
reliability of the exam,
b) index of difficulty
(i.e., a value showing
the proportion of
candidates answering an
item correctly), c)
discrimination indices
(i.e., values showing
how well an item
succeeds in
distinguishing highly
competent from less
competent candidates)
and d) distractor
analysis (i.e., the
frequency with which
each option of a
particular test question
is chosen). This
information combined
with more sophisticated
statistical measures
enable test developers
and item writers to
evaluate item and task
performance ensuring
overall test validity
and reliability.
See
also: ELT
News KPG corner
THE INTER-RATER
RELIABILITY PROJECT
The on-going study on inter-rater reliability (i.e., agreement between raters on
scores assigned to papers) in the KPG script rating process is carried out as a
means of investigating the effectiveness of the instruments subservient to the
process employed by the KPG test developers. These instruments are: a) the
rating grid together with the assessment criteria, b) the script raters training
material and training seminars and c) the on-the-spot consultancy to the raters
by KPG experts and test developers. The aim of the particular project is to look
into the factors affecting the rating process in the specific situation and
examine the extent to which these factors contribute to higher inter-rater
reliability. The data consists of the marks given by different raters to A1+A2,
B1, B2 and C1 level scripts of candidates who have taken the KPG exam in English
and is being obtained at the Athens Script Rating Centre.
THE
TEST-TAKING STRATEGIES
PROJECT
The KPG
test-taking strategies
project aims at
investigating the
test-taking strategies,
i.e. the techniques or
‘tricks’ consciously
used by KPG test takers
so that they can respond
to KPG test requirements
and whether these can be
successfully taught.
Questionnaires are the
basic tools designed
especially for this
project, and their
development was a
complicated procedure
involving different
groups of research
participants at various
stages of the research
process, including the
research subjects.
Research results have
already been published
and presented in
numerous conferences.
See
also: ELT
News KPG corner
