by Kristie Dukewich & Carmen Larsen, Curriculum Consultants at TCDC
Before I taught my first course, I thought that preparing an exam for students was going to be easier than preparing a class lecture. I expected that exam week was going to be my easiest week in terms of course prep. I did not expect that I’d have to develop a whole new set of skills related to designing assessment. So much for expectations.
Designing effective exams has turned out to be much more complicated and time-consuming than I first imagined. There are so many variables to consider in designing an effective exam, including:
- writing questions that are clear and concise using simple, accessible language
- selecting specific topics to assess (and deciding which topics can be left out)
- assessing a range of cognitive complexities
One of the aspects of exam design that I have wrestled with a lot over the years is cognitive complexity. How do I write exam questions that are challenging to students, but not too challenging? How can I assess higher-order thinking without destroying students’ confidence and motivation with overly-difficult questions?
___________________________________________________
Are you interested in learning more about designing exams and evaluating their effectiveness? Sign-up for our workshop.
___________________________________________________
Learning taxonomies can be incredibly valuable for helping us think about the issue of cognitive complexity in assessments. A variety of learning taxonomies have been developed that attempt to categorize levels of knowledge and/or ability. Bloom’s Taxonomy (see Krathwohl, 2002 for an overview) is probably the most well-known. It identifies a range of learner skills and abilities that are anchored by a foundation of basic knowledge of a topic. The model is hierarchical, with the level of understanding (or cognitive processing) becoming increasingly complex as we move through the 6 stages from the basic ability to recall information to the ability to analyze and evaluate it and then finally using it to create something new. This taxonomy can be very helpful when considering the types of questions we will use in our exams.
However, one of the problems with Bloom’s Taxonomy is that it implies the development of knowledge and skills in a linear manner. The reality is that when students are constructing knowledge, they tend to skip around – from applying knowledge, to analyzing their application, to evaluating the effectiveness of the application, then back to analysis using a different framework or approach.
Moreover, the higher-order cognitive skills don’t necessarily reflect cognitive complexity. For example, application doesn’t need to be cognitively complex. Your students might be really good at applying MLA formatting, but you wouldn’t call that higher-order thinking.
Another challenge when using this taxonomy is we cannot presume that asking a higher-order question will elicit a cognitively complex response from our students. In fact, asking our students to evaluate something may lead to a less cognitively complex answer than if we ask them to analyze it even though evaluation is higher on the taxonomy.
Remember: Who wrote In the Skin of a Lion?
Understand: Describe the subject matter of In the Skin of a Lion.
Apply: Relate the story of In the Skin of a Lion to a current event.
Analyze: Describe Ondaatje’s use of symbolism in In the Skin of a Lion.
Evaluate: What is your opinion of In the Skin of a Lion?
Create: Building on the themes found in In the Skin of a Lion, develop a poster, video, or podcast that explores the lives of Canadian immigrants today.
**********
A less well-known taxonomy that does address cognitive complexity and how learners construct knowledge is Biggs’ Structure of the Observed Learning Outcome (SOLO) model (Biggs & Collis, 1982; Biggs & Collis, 1989). This model outlines 5 stages of understanding or ascending structural complexity.
In the prestructural stage, learners have no understanding of a topic. They can demonstrate an understanding of one aspect of the topic in the unistructural stage, and several relevant but relatively independent ideas about the topic in the multistructural stage. In the relational stage, learners are able to synthesize and integrate multiple ideas. And finally, in the extended abstract stage, they are able to generalize what they’ve learned to new situations or make new connections.
Many educators prefer this model because it allows them to more easily assess the level of complexity of both their exam questions and their students’ answers. It also allows students to reflect on their level of understanding and the quality of their responses to exam questions.
Unistructural: Who wrote In the Skin of a Lion?
Multistructural: Give examples of 2-3 literary devices used in In the Skin of a Lion. Relational: Relate the story of In the Skin of a Lion to a current event.
Extended abstract: If Ondaatje had set In the Skin of a Lion in Vancouver in 2018, how would the story be different?
************
If you’d like to learn more about designing effective exams, please join us on Friday, March 16th from 9am-10:45am for our workshop, Designing Exams and Evaluating their Effectiveness. During the workshop, participants will be introduced to ways of designing exam items that target different levels of cognitive complexity, removing extraneous exam elements, and evaluating how effectively an exam assesses students’ achievement of course learning outcomes. We will also look at how to interpret and use item analysis reports to improve the quality of multiple choice questions.
