The Impact of Test Blueprint Transparency of Hematology Course on Students’ Evaluation and Final Grades

Corresponding Author:

Fahad AS Aleidan
College of Medicine, King Saud Bin
Abdulaziz University for Health Sciences
Riyadh, Saudi Arabia
E-mail: faleidan@gmail.com

Abstract

Abstract

Background: Test blueprint, sometimes referred as table of specifications, is rarely used in the construction of traditional summative assessments in medical schools.

Objectives: To assess the impact of releasing test blueprints on students’ attitudes towards the Hematology block evaluation.

Methods: The test blueprints for the mid-block and final written examinations were released and well explained at the start of hematology block by the block chief coordinator in 2016. These students from the 2016 cohort serve as the “after blueprint released group”, while students in 2015 cohort serve as “before blueprint released group”. The end of block evaluation was used for the mid-block and final examinations, using a 5-point Likert scale.

Result: In the mid-block examination, there was significant difference in the number (%) of students rating the course evaluation, with a comprehensive coverage of the content taught [119 (82.1) versus 83 (62.9), p<0.001] and with overall fairness of the examination [120 (82.8) versus 90 (68.2), p=0.005] between the after test blueprint released group and the before released group. Similarly, in the final examination, there was significant difference in the number (%) of students rating the course evaluation with a comprehensive coverage of the content taught [127 (87.6) vs 80 (60.6), p<0.001] and with overall fairness of the examination [123 (84.8) versus 95 (72.0), p=0.009]. The mean (± SD) mark of the after test blueprint released group was significantly higher than the before test blueprint released group [88.7 (± 8.50) versus 82.95 (± 9.75), p<0.0001].

Conclusion: The release of the test blueprint at the beginning of the hematology block improved the students’ perception of the fairness and comprehensiveness of the final examination and also led to their higher mean level of performance.

Keywords

Evaluation; Test blueprint; Medical education; Student; Grade

Introduction

At King Saud Bin Abdulaziz University for Health Sciences (KSAU-HS) College of Medicine (COM) model of PBL is adapted from the University of Sydney PBL model, and implemented according to the Saudi Arabian socio-cultural context. Hematology is the fourth block in the first year of basic sciences, phase-II. This block consists of six weeks and includes five cases (anemia, chronic lymphocytic leukemia, bleeding disorder, thalassemia, and thrombosis) and specific learning objectives for each case. Mid-block and end-block written exams comprising of Multiple-Choice Questions (MCQs) are used as a summative assessment tool for students to achieve their final grades. Other assessment methods are also used for different instructions, including Objective Structured Clinical Examination (OSCE), and Objective Structured Practical Examination (OSPE) [1].

The occurrence of mismatching between the content examined and the material assessed at the end of a course is frequently realized and perceived. This lack of coherence leads to an assessment that fails to provide evidence from which instructors can make valid judgements about students’ progress and grades [2]. The development and utilization of constructed test blueprints is one of the strategies to be adapted to mitigate mismatching. The test blueprints can assist the instructors to align the amount of class time spent on individual learning objective delivery with the cognitive level, thereby allowing instructors to identify the types and numbers of items needed to be included in their exams [3]. However, the influence of test blueprint transparency on students’ evaluations and grades has not been extensively studied. The aim of this study was to assess the influence of releasing test blueprints on students’ evaluation and grades.

Methods

At the end of the block, a self-administered questionnaire was used to gain student feedback and attitudes towards the instructional effectiveness, stimulation, subject relevance, amount learned from the cases and quality and difficulty of mid-block and final written examinations. Students rated items on a 5-point likert-type scale anonymously, where 1=poor and 5=excellent. Space was also provided for open-ended comments to the questions “what did you like most about this block?” and “what would you like to be changed in this block?” This questionnaire was developed by the department of medical education, and a group of medical educators assured content validity. Reliability,measured using an alpha coefficient, was 0.914.

The test blueprints for the mid-block and final written examinations were released and well explained at the start of the hematology block by the block chief coordinator (Tables 1, 2) in 2016, and these students serve as the “after blueprint released group”, while students in the 2015 cohort serve as “before blueprint released group”.

Table 1: The assessment blueprint for the mid-block written examination of hematology block.

Table 2: The assessment blueprint for the final written examination of hematology block.

At the end of the block, students were asked to evaluate the mid-block and final examinations using the 5-point likert scale comprising the following items: Comprehensive coverage of the content taught; Quality of MCQs; Difficulty level of examinations; And overall fairness of examinations.

Data analysis

students’ feedback data were entered into an excel spreadsheet. The mean responses were calculated for all evaluation questions. Open-ended comments were analyzed qualitatively to explore the content of commentaries, to compare and contrast the strengths and weaknesses of both the block and PBL cases in terms of relevance, stimulation, and amount of knowledge learned as perceived by the students, and more importantly, the tutors. The end of block questionnaires for mid-course and final examinations before and after test blueprint released were compared and analyzed using the Z score test for two population proportions. Students’ grades as mean mark (± SD) were analyzed and compared using unpaired t-test. All tests were two-sided and a P value <0.05 was considered significant.

The Statistical Package for the Social Sciences (SPSS) software (USA, version 24) was used to perform the statistical analysis.

Results

Out of 298 students, 129 from each of the two batches answered and returned the questionnaires, with an overall response rate of 93%. Before the blueprint was released 132 out of 140 students answered and returned the questionnaires, with a response rate of 94%. After the blueprint was released, 145 students out of 158 responded and answered the questionnaires, with a response rate of 92%. The high rate of response helps justify the accuracy of the results to be presented in the study. Descriptions of students’ end of hematology course evaluation in term of block organization, PBL cases, and performance of PBL members before and after test blueprint released, are presented in Table 3.

Table 3: Students’ end of hematology block evaluation before and after releasing blueprint.

In the mid-block examination, there was significant difference in the number (%) of students rating the course evaluation with a comprehensive coverage of the content taught [119 (82.1) versus 83 (62.9), p<0.001] and with the overall fairness of the examination [120 (82.8) versus 90 (68.2), p=0.005] between the after test blueprint released and before released groups. In the final examination, there was significant difference in the number (%) of students rating the course evaluation with a comprehensive coverage of the content taught (127 (87.6) versus 80 (60.6), p <.001) and with the overall fairness of the examination (123 (84.8) versus 95 (72.0), p=0.009) (Table 4).

Table 4: Student’s evaluation for the mid-block and final exam before and after blueprint released.

The after test blueprint released group achieved higher average grades than the before test blueprint released group (B+ versus B). The mean (± SD) mark of the after test blueprint released group was higher than the before test blueprint released group (88.7 (± 8.50) versus 82.95 (± 9.75), p<0.0001) (Table 5).

Table 5: Students’ final grades before and after releasing hematology block blueprint.

Discussion

This study has shown that releasing the test blueprint at the beginning of the block, along with an in-depth discussion of the course and its objectives with the course director, had a positive impact on how students perceived the fairness of the mid-and end-block examination. Additionally, it positively affected the students’ perception of the comprehensiveness of the material taught and learned in the block. The release group also performed better on the midand end-block examinations than the pre-release cohort of students.

While little had changed in the block between the two years, a perceived weakness of the study is not having a true control group other than the cohort of students who participated in the block the year prior to release of the test blueprint [4].

The other difference from a previous study of making test blueprints available to students at the University of Calgary, was that the hematology block test blueprint only included the types of instruction from which the test questions would be drawn as opposed to a more precise listing of the actual subject matter of the question [5]. Nevertheless, the students in the release group not only felt more positive towards the mid-and end-block examinations but their mean level of performance was higher than the pre-release cohort, as noted above.

A well-constructed blueprint test offers content validity to the process of evaluation [6]. In addition to this psychometric benefit, a wellconstructed test blueprint also has practical advantages for all involved in the educational experience, including the course chair, the evaluation coordinator, and course instructors. However, while these advantages are generally accepted, the practice of releasing test blueprints per se is less well accepted. The argument against providing students with a test blueprint because it may make the evaluation easier for them appears to be unfounded; however, another argument suggests that it may drive students to ‘strategic learning’ [7] a consequence of which is a risk that their knowledge base may be less ‘rounded’.

The limitations of our study include its design of comparing the current hematology block to previous year’s block, which may present a selection bias. As mentioned above, blueprint release may drive students to strategic learning. Lack of previous experience in our college of medicine curricular employed.

Conclusion

The release of the test blueprint at the beginning of the hematology block improved the students’ perception of the fairness and comprehensiveness of the final examination and also led to their higher mean level of performance.

References

1. Al-Eidan FAS. Design of the hematopoietic system block in an integrated problem-based curriculum. J Appl Hem 3, 114-118 (2012)
2. Brookhart NE. Teaching about communicating assessment results and grading. Educat Meas Iss Prac 18, 5-13 (1999).
3. Anderson LW, Krathwohl DR, Airasian PW, et al. A taxonomy for learning, teaching and assessing: A revision of bloom’s taxonomy of educational objective (Abridged Ed.). New York: Longman (2001).
4. Al-Eidan F, Baig LA, Magzoub ME, et al. Reliability and validity of the faculty evaluation instrument used at King Saud Bin Abdulaziz University for Health Sciences: Results from the hematology course. JPMA 66,  453-457 (2016).
5. McLaughlin K, Coderre S, Woloschuk W, et al. Does blueprint publication affect students’ perception of validity of the evaluation process? Adv Heal Sci Edu 10, 15-22 (2005).
6. Bordage G. Content validation of key features on a national examination of clinical decision-making skills. Acad Med 70, 276-281 (1995).
7. Newble DI. Learning styles and approaches: Implications for medical education. Med Edu 20, 162-175 (1986).