ELT News, April 2009

The KPG Exams From The Point Of View Of The Candidate

There are more than a dozen foreign language certification systems that the Greek ELT community is familiar with and each one promotes the quality that it believes will make it most attractive to its potential clients , or the feature that best characterizes it. A claim repeatedly made by KPG is that the state exams are the only ones which assess language in use by people whose interests, experiences, sociolinguistic awareness and literacy are seriously taken into account. To support this claim, we would like to present our readers with findings from one out of several research projects carried out at the Research Centre for English of the University of Athens (http://rcel.enl.uoa.gr/), concerning the KPG exams. The project directed by Jenny Liontou, under the supervision of Bessie Dendrinos, aims at investigating candidates' views on various aspects of the exams, one of them being reader variables and how these affect text difficulty and comprehensibility. A variety of research tools are used for this purpose, including questionnaires which seek to find out how difficult the candidates think that the test papers are.

Our findings are the result of quantitative analysis of the responses to especially designed questionnaires, distributed evenly throughout Greece .  Each questionnaire takes only 5-7 minutes to fill in, and candidates are asked to complete it before they leave the exam room after each exam administration, on a voluntary basis.  Preliminary questions seek information regarding candidates' age, sex, number of years they have been studying English and whether they have come especially prepared for this exam. The rest of the questions (35 in total) seek to find out the candidates' opinion about the difficulty of the exam as a whole, about each test paper and about the activities in each test paper separately. Also, there are questions about the rubrics, the candidates' familiarity with the topic and genre of each text, and about time (i.e., if time was sufficient for them to complete the test paper).

  A minimum of 10%, and no fewer than 500 questionnaires, are analysed after each exam administration, and data are electronically recorded in a specially designed database. The findings to be discussed presently (see Table 1 below) are derived from a total of 7250 questionnaires: 4750 are the responses to questionnaires designed for B2 level candidates and 2500 responses of C1 level candidates.

With regard to the KPG C1-level candidates' profile, age values in all four exam periods ranged from 12 to 48 years. Although age values follow no normal distribution, most occur within the range of 16-24 years, as was the case for B2 candidates, with 16 being the most frequent figure. As with the B2 level candidates, women taking part in the C1 exam outnumber men in all four exam periods. The majority of C1 candidates seem to have been learning English for 7-9 years before sitting for the exam. However, as opposed to the B2 findings, a lower percentage of candidates sitting for the C1 exam have stated that they have undergone some kind of special preparation before sitting for the specific exam i.e.  less than half (45-47%).

Candidates' views About the exams

• Topic interest

Candidates find the text topics interesting most of the time, but how interesting each text is for each candidate depends on his/her age. Young candidates like different texts than older candidates. In other words, as expected by the researchers, the different age groups like different texts.

• Mediation

Performing as a mediator is something everyone seems to like; that is, to read a Greek text (e.g. a newspaper article, a poster, leaflet, etc.) and relay some of the information or ideas it contains either orally (in the case of the oral test) or in writing (in the case of the writing test).

• Degree of test/ text difficulty

Our data shows that age is a factor which has an important impact on the candidates' opinion regarding the degree of difficulty of a text and comprehension activities in particular. The degree of difficulty is also strongly related to their familiarity with the text topic. The more acquainted they are with the topic and the more interesting candidates think that a text is, the easier they find it, regardless of its complex structure or its range of vocabulary. But there is one more finding which confirms our initial assumptions. That is, the younger the candidate, the more difficult s/he thinks that the exam is, and the older the candidate the more likely s/he is to enjoy listening or reading about topics of social concern, such as about environmental issues. Furthermore, statistical analysis of our data shows that there is a strong correlation between the number of years the candidate has been studying English and the familiarity s/he has with the text topic and genre. On the whole, candidates who have studied English for a longer period of time find the language of the text easier and the test activities less difficult.

With regard to the difficulty of the reading comprehension test in particular, the majority of the B2 level candidates (79,5-86,7%) believe that the items are ‘slightly' or ‘fairly difficult', whereas those that believe that the test paper is ‘very easy' or ‘very difficult' are fewer. This finding indicates that the exams seem to coincide with the average candidate's expectations and to meet the English Team's set objectives. When asked to express their opinion about the rubrics, most candidates(79-86,2%).said that they were easy to understand. However, when asked about the degree of text and corresponding item difficulty, their responses varied from one exam period to another, and the variation is relatively significant. The findings are extremely useful for the English test design team, especially when these are considered together with the results of the a posteriori item analysis undertaken after every exam administration. Actually, it is exactly this kind of information which turns the candidates' questionnaires into important research tools, since they enable the researchers to spot the diversity of candidates' views.  What is more, this kind of information can be used to re-interpret reactions to the exam and to cross-check them against other quality control exam procedures.

The opinions of the C1 level candidates are similar to those of the B2 level candidates.  In terms of the reading comprehension test difficulty, for example, most candidates (77,1-88,1%) find that it is 'slightly' or 'fairly difficult', whereas the majority (86,3-89,9%) say that the rubrics were easy to understand. The level of difficulty of the texts and their corresponding questions also varies to a certain degree from one exam period to another.

Conclusion

Our systematic analysis of candidates' views brings to the surface that there are some general tendencies worth mentioning: Regardless of the language level, candidates show a clear preference to familiar, everyday issues such as the benefits of physical exercise (top choice for B2 level candidates) or famous people (top choice for C1 level candidates). Topics that have to do with the Greek reality are next (Greek traditions, protecting the environment of Greece, etc.).  The least interesting subject areas seem to be the ones that the candidates are not familiar with or contain distinctive information (special dietary habits, contemporary European history, etc.) The above information is of vital importance given the fact that topic preference influences candidates' performance. This means that the KPG item writers are thus able to choose those topics that match the interests and needs of the candidates.

Generally speaking, unlike international exams, the Greek state exams cater to the needs of Greek users of English and not to the needs of an international clientele. International exams cannot possibly take the needs, interests, experiences, and other such factors into account because it is not cost affective to tailor the tests to local clients For KPG designing exams for the people for whom the exams are addressed is one of the most important aims of the university test design teams and of the Ministry of Education, which administers these exams and guarantees security and fairness of results.

Bessie Dendrinos & Jenny Liontou