Investigating the Reliability of Cet- Reading Test Using Winsteps Rasch Model

First 2-3 words of Title        1

[pic 1]

Final Assignment

17/11/2016

First 2-3 words of Title        2

Investigating the reliability of CET- Reading Test using Winsteps Rasch Model

Wong Hui Min

First 2-3 words of Title        3

Introduction

The College English Test (CET) serves as a benchmark to determine whether the Chinese undergraduates have mastered the language objectives outlined in the National College English Teaching Syllabus (NCETS). Aligned to the syllabus, the CET comprises of three components: Band 4 (CET-4), Band 6 (CET-6), and the CET-Spoken English Test (CET-SET) which are administered by the National College English Testing Committee. Both the CET-4 and CET-6 are conducted at the end of each semester, in June and December respectively. The examination involves five components: Listening and Reading Comprehension, Vocabulary and Structure, Error Correction and Writing and the duration is 2 hours and 20 minutes.

For the purpose of this paper, the focus will be on the Reading Assessment of the Band 6.The weightage of the Reading comprehension is 35%. Of which, 25% involves extensive reading which involves the testing of reading comprehension of essays with selective genres, cloze passages and short answer questions. The remaining 10% will be on skimming and scanning. The test is divided into 6 parts. Part 1 comprises of 5 different paragraphs where pupils have to read and match them to the sentence which summarize the main idea from a list of options. The texts provided are mainly business reports reporting on the profits or certain business decisions made by the management. Part 2 consists again a business report explaining the importance and the implications of the success of manager-trainers in their research and partnership with the business fraternity. Pupils are required to match the 6 blanks with the most coherent paragraph consisting of 3 sentences from the list of 8 options. This is testing on pupils’ skill of inference and making sense of the text. Part 3 consists of a reading text and 6 multiple choice question (MCQ) testing on pupils’ skill of reading for details. The text provided is

First 2-3 words of Title        4

once again a business report which describes the transition of a certain sports brand from manufacturing shoes to a more diverse product selection. Part 4 consists of a cloze passage of 10 blanks in an MCQ format which requires test-takers to choose from 4 options the most appropriate word. One may argue that this item not only is assessing reading proficiency but also vocabulary. Messick(1996) explains the this could contribute to Construct irrelevant variance where the assessment assess irrelevant skills such as in this case, vocabulary , which may interferes with the test-takers’ performance. Part 5 includes a cloze passage of 10 blanks in which the test-takers have to fill in the blanks with a word of their own choice. The answers span over a wide array of word class and test-takers not only have to decipher the meaning of the text but employ other linguistic knowledge such as syntax to be able to get the correct answer. Part 6 is an editing exercise with 12 questions where test-takers have to differentiate the sentences with the redundant words and the correct ones. This requires syntax proficiency and comprehension of the text is vital as the redundant words may not be grammatically wrong but do not fit in with the text.

The paper aims to investigate the reliability of CET- Reading Test using Winsteps-Rasch Model. Bachman & Palmer (1996) asserts that in order for a test to be valid, its prerequisite is reliability. In a performance assessment, test- takers scores are the product of the dynamics of many factors and therefore it is insufficient to assess the reliability of the assessment based only on inter-rater or intra-rater consistency. The Rasch measurement is conceptualized as simple rectangular data units of persons and test items using the program, Winsteps. Additionally, Item types such as dichotomous, multiple-choice, and multiple rating-scale and partial credit items could be pooled

First 2-3 words of Title        5

together, forming one analysis. Thus, paired comparisons and rank-order data could also be analyzed to facilitate communication and exploration where the relationship between items and persons could be further examined. Therefore, by employing Winsteps, the usefulness of the CET could be easily evaluated and decisions on the construction of more effective tests could be made quickly.

Literature Review

In Alderson (2000)’s schema theory, he re-iterates the influence of test-takers’ prior and topical knowledge in top-down processing which will give them an advantage. In the perspective that the influence of topical knowledge encompasses all comprehension, Alderson (2000) suggests that this variable should be harnessed to facilitate performance rather than allowing its absence to inhibit performance. Alternatively, Douglas (2000) perceived that in certain reading assessments especially in academic reading, subject knowledge and register should be part of the construct. In view of the CET, the subject matter is focused on formal business reporting. It seemed to advantaged /business majors where they are familiar with the content and more exposed to the register, thus more knowledgeable in the specialized vocabulary in context and text structure. However, the CET is not just for business majors, hence there could be bias which give rise to construct irrelevant variance. Furthermore, Alderson (2000) emphasize the transference of Linguistic knowledge of L1models to the L2f literacy and the need to refine the construct to assess the relevant skills in the L2.

Thus, considerations such as content relevance and coverage should be made to ensure validity in second language performance assessment. Bachman (1990) defines content validity as content relevance and content coverage. Content relevance refers to the matching of the linguistic demands of the tasks to targeted language ability, thus

First 2-3 words of Title        6

acting as an indicator to whether the test taker has achieved mastery in the specified ability domain and the test method facets. On the other hand, content coverage deals with the craftsmanship of the test to sufficiently validate the performance and Spolsky & Bachman(1991) suggest employing random selection of sample texts to achieve this. However, this could be a challenge as test takers come from diverse backgrounds and have extensively ranging language needs.