Assessing C1 KPG candidates’ pragmatic competence in written tasks: towards the design of task-specific rating scales (2009)
Karatza Styliani (Full Paper)
Master's Dissertation
Department of Language and Linguistics
Faculty of English Studies
School of Philosophy
National and Kapodistrian University of Athens

Many authors have occupied themselves in studying the development of learners’ pragmatic competence and in exploring the field of language testing and assessment. Nevertheless, little work has been done with regard to the assessment of learners’ pragmatic competence and the design of effective instruments for pragmatic performance measurement. The present dissertation attempts the design and application of a task-specific rating scale for the measurement of C1 Kratiko Pistopiitiko Glossomathias (KPG) candidates’ pragmatic competence. This research project views pragmatic competence as EFL learners’ ability to produce texts which constitute instantiations of particular genres, by selecting linguistic structures which are conventionally attributed to the register and style of the intended text type (genre).

My study has been based on the assumption that task-specific rating scales could assist script raters during the use of the first evaluation criterion of KPG Module 2 activities and increase reliability of scoring. On this premise, this study sets out to develop a methodology which will enable the design of pragmatic competence task-specific rating scales for KPG tasks. After textual analysis of authentic texts and validation of results by expert judges, a task-specific rating scale is devised on the basis of an Index of Pragmatic Performance (IPP)

The devised scale is deployed for the assessment of candidates’ pragmatic performance in a total of 42 excellent and medium C1 level KPG scripts. The analysis of the findings follows a qualitative approach combined with occasional reference to quantitative data. 
After the examination of data, it is revealed that KPG candidates’ pragmatic performance can range from poor to excellent. A finding worth mentioning is that all the pragmatic genre-specific features of school newspaper opinion articles were detected in C1 KPG scripts. Pragmatic deficiencies were traced in both excellent and medium KPG scripts. A strong positive correlation was detected between pragmatic performance scores (PPS) and language performance scores (LPS). Finally, the usefulness of KPG task-specific rating scales is discussed with reference to script raters’ training and development of exam-preparation materials.