Online assessment
Introduction
We noted in Chapter 3 the close relationship between the design of online learning activities and assessment tasks, and the advantages of merging activities and assessment where this is possible. Hence, we have already considered some ideas that are important in online assessment, including the alignment of objectives, activities and assessment, and the role of feedback in supporting and extending learning opportunities. We will now explore in more detail some of the multiple reasons and opportunities for assessing online, and some of the challenges you might face.
Making decisions about whether, when or how to use online assessment in your teaching requires a clear focus on the nature and purposes of assessment and on the basic principles of assessment design. It also requires you to know about the affordances of the online technologies that are available to you, which you considered in Chapter 2. We will begin by reviewing some general characteristics of assessment in higher education, then introduce some of the common uses of the online environment for assessment, and identify some opportunities and challenges of these uses that you might consider. We will also address ways of implementing assessment practices online and deal with some of the support, management and administrative issues that arise if you are planning online assessment.
We proceed from the assumption that assessment should only occur online if this benefits, or does not inhibit, student learning. The latter point is important because online assessment offers a number of administrative benefits (for the student and/or the teacher), as well as benefits to learning, and it is important to consider these. However, we support the view that an administrative benefit should not be accompanied by a cost in terms of student learning. In this topic we focus mainly on considerations about assessment which relate to student learning.
We begin by reviewing some key understandings about the nature and purposes of assessment to provide a context for the kinds of opportunities offered by online assessment.
Why assess?
When we think of assessment we often think of examinations and assignments, of grading the performance of students in order to decide if they should ‘pass’ a particular subject or part of it. But assessment reaches beyond grading, and online assessment helps us to extend that reach, not only by facilitating the assessment process but also by opening up possibilities that are not available in the face-to-face environment.
Ramsden (2003) summarises some of the important contemporary themes relating to the role of assessment in higher education. As a central assumption, he draws on the view of assessment expressed by Rowntree (1977) which we mentioned in Chapter 1. This suggests that assessment is an interaction which is aimed, to some extent, at knowing another person. The view that assessment is about more than the measurement of performance has become influential in recent years. Serafini (2004) identified three paradigms of assessment: assessment as measurement; followed historically by assessment as procedure; and thirdly, assessment as inquiry. The idea of assessment as inquiry has led to a focus on its value for learning, rather than being ‘primarily about the allocation of rewards and punishments’ (Ramsden, 2003, p. 180). Hence, the distinction is often made between formative assessment for learning (e.g., Boud, 1995, 2007; Carless, 2007), assessment of learning (summative assessment), and assessment as learning, ‘when students personally monitor what they are learning and use the feedback from this monitoring to make adjustments, adaptations, and even major changes in what they understand … [which] is the ultimate goal where students are their own best assessors’ (Earl, 2003, p. 25).
The aspects of feedback that we considered in Chapter 3 illustrate characteristics of formative assessment. Gibbs, Habeshaw and Habeshaw (1988, p. 7) explained the formative and summative components of assessment as follows:
Assessing is a general term used to describe all those activities and processes involved in judging performance. Assessing can be summative or formative: summative assessment is concerned with a final judgement of performance; formative assessment is concerned with the improvement of performance. In broad terms marking and grading involve summative assessment while reviewing and giving feedback involve formative assessment.
For example …
You may want to know how learners are progressing towards the outcomes specified by the learning objectives so you can target your teaching effectively at either group or individual level. You can make a formative assessment of how learners are responding to the activities you have designed, by looking at their input and/or asking them questions about problems they may have. You will then be able to give learners feedback, as we discussed in Chapter 3, which helps them get over hurdles and resolve difficulties. As well as supporting their learning this will give you ideas about teaching strategies you might implement to address any problems they are having.
If you want evidence of students’ learning in a subject in order to give them a score for their achievements, you might use a variety of assessment tasks (some of which we will consider in this chapter), including formal examinations. The learning objectives often determine the most appropriate form of the assessment tasks. The combined grades from these tasks would then be used for a final judgement of performance (summative assessment), usually as students complete their study of a subject.
Within these two purposes of assessment, further distinctions can be made. Among the many reasons for assessment Crooks (1988) listed the following as relevant in higher education:
3. focusing learning – the ‘hidden curriculum’;
4. consolidating and structuring learning;
5. guiding and correcting learning;
6. determining readiness to proceed;
Your own reasons for assessing students may vary from those listed, but note how all but the last of these are aspects of either formative assessment (which deal with guiding the learning of students through providing support and feedback) or summative assessment (those that are concerned with summarising and confirming past achievement). Because the formative role of assessment has most direct influence on the process of student learning, making sure you have considered this carefully will give you more options to consider about how you might assess online.
For example …
Contributions to online discussions provide opportunities for feedback from you or from other students in the class.
Online formative assessment quizzes are easy for students to undertake, in their own time, as many times as they need to, offering immediate feedback which you have designed as part of the test.
A part of formative assessment involves identifying students’ prior knowledge and skills in order to clarify the aspects of their performance which you (and they) will need to focus on. Diagnostic assessment is often used at the beginning of a teaching period for this purpose and to help you find out more about the characteristics of your students so that you can provide the support they need to achieve the intended learning outcomes.
For example …
set a pre-test with incorporated feedback (which is matched with a later post-test to identify the learning that has occurred);
ask students to write a letter or reflection about themselves and their learning;
ask students to complete a self assessment task in which they evaluate their past learning; or
provide a brief questionnaire on issues related to students’ goals, expectations, potential problems, etc.
Note: The use of audience response systems or networked tablet computers in a classroom-enhanced context provides excellent opportunities for diagnostic assessment and formative guidance and feedback at a class level.
You might be concerned about the extra workload that these different forms of assessment could have for you, but there are many ways of managing assessment and the online environment can help you to do this. Distributing the workload is one option. The question of ‘who assesses?’ has quite a few implications for assessment in the online environment.
Who assesses?
The traditional answer to the above question is you (the teacher), or tutors, or markers whom you pay to do the job. This may be relevant for much of the summative assessment of your students but it ignores important opportunities in assessment for learning. Identifying others who are able to contribute to assessment can help you to deal with workload, help students to take responsibility for their own learning, and has the potential to bring in new perspectives, e.g., from experts. You may already be implementing self or peer assessment which raises the possibility of encouraging students to be assessors and opening up to them the additional learning possibilities associated with the assessment role.
Self assessment
Self assessment encourages students to become active, independent learners and contribute to the generation of feedback for themselves, helping to prepare them for life after university. Boud (1991, p. 5) states that self assessment is about the involvement of students in:
identifying standards and/or criteria to apply to their work and
making judgements about the extent to which they have met these criteria and standards.
Self assessment means more than students grading their own work; it means involving them in the processes of determining what is good work in any given situation. They are required to consider what are the characteristics of, say, a good essay or practical report and to apply this to their own work.
As we have already noted, aspects of the online environment that support reflection, including e-portfolios, blogs, and online discussion, can be very helpful in encouraging self assessment.
Peer assessment
Roberts (2006, p. 6) defines peer assessment as ‘the process of having the learners critically reflect upon, and perhaps suggest grades for, the learning of their peers’.
Although there are valuable learning opportunities in asking students to assess the work of their peers, there are common problems associated with summative peer assessment. These include collusion, reluctance to participate and friendship bias, as well as effects of gender, age, ability or ethnicity. Strategies such as requiring students to justify ratings, and explicitly linking criteria with ratings, can address a number of these problems (Falchikov, 2005). When using peer assessment summatively it is better to use an overall global mark with well-understood criteria, and to involve students in discussions about criteria, than to ask them to rate many individual dimensions (Falchikov, 2005).
A number of software programs have been specifically designed to implement structured peer assessment online and support the provision of feedback and grades by students. See Tucker, Fermelis and Palmer (2009), Raban and Litchfield (2007) or Sung, Chang, Chiou and Hou (2005) for some recent examples. You would be unlikely to consider one of these if you were embarking on online assessment unless a similar system already exists at your institution. At a simpler level, your learning management system may provide customisable grading forms which you can use to support reliability and validity in the context of peer assessment.
Other assessors
Other potential assessors include mentors (formal or informal) and clinical or workplace supervisors or experts. Feedback from such people can increase authenticity of assessment and the online environment facilitates their input.
For example …
By inviting an expert (from anywhere in the world) to moderate an online discussion on a particular topic, students are able to benefit from ‘real-world’ experience and receive useful feedback. While this will relieve you of the demands of responding to students during this period you will still need to monitor the discussion in the context of the subject as a whole. The assessment task may include designing questions for the expert which could contribute to summative assessment.
Clinical or workplace supervisors will provide invaluable formative assessment for students. As well as structured assessment events, make sure your students take advantage of the informal assessment that can be provided by such supervisors.
Some aspects of assessment design
In this section we will consider some important aspects of assessment design. These apply whether or not you are assessing online, but we will focus on those which have particular relevance in the online environment. We addressed some of them in Chapter 3 when we considered the alignment of objectives, activities and assessment, including the benefits of merging learning tasks and assessment tasks (Oliver & Herrington, 2001), and when we noted the value of authentic learning tasks. We cover these points again here because of their importance to assessment. As in all design for online learning and assessment, an awareness of student characteristics and the learning and teaching context is an essential starting point.
Accommodating the learning and teaching context
Accommodating the learning and teaching context in assessment involves considering factors such as:
the level of study (to determine the extent of guidance and structure that you need to provide);
whether your assessment tasks accommodate different learning styles and abilities;
whether the workload is appropriate; and
whether the assessment is fair and offers equal opportunity to all students. This may include determining whether you have provided sufficient choice to accommodate different backgrounds and experiences (including different cultural backgrounds).
Depending on the learning outcome that is being assessed, in many circumstances a valuable way of acknowledging the student’s context is to incorporate it in the assessment task. This has added advantages in terms of authenticity and it reduces the likelihood of plagiarism because the task is individualised.
Depending on the nature of the assessment, you may also need to consider factors such as access, costs and ethical issues.
Aligning objectives, activities and assessment
Whether or not learning activities are formally included as part of the summative assessment, they should always underpin assessment in some way to allow students to practise and demonstrate the kind of learning specified by the learning objectives. This alignment then also needs to be evident in the summative assessment task.
Designing authentic assessment tasks
Authentic assessment places value on both the process and the product of learning and provides added motivational benefits for students in terms of interest and relevance (avoiding the traditional assignment plus examination combination!). It is particularly applicable to online assessment because the environment offers affordances and tools to complete tasks in authentic ways and the process, as well as the product, is visible to the assessor.
Mathur and Murray (2006, p. 256) have developed a useful checklist to determine the authenticity of an online assessment strategy. See Table 4.1.
Table 4.1
Checklist to determine the authenticity of an online assessment strategy
Source: Mather and Murray, 2006, p. 256.
While this list is not specific to the online environment, it provides a way to ensure the authenticity of your assessment design.
Designing group assessment
Group work is increasingly valued in higher education because it allows students to practise the kind of team-based tasks that they will encounter in the workplace; hence, it lends itself to authentic assessment. These activities provide immediate opportunities for formative peer assessment. This may be implicit in the design of the activity (through students negotiating), or you might explicitly request it. However, students often dislike summative assessment of group work when it does not recognise their individual contributions. Drawing on previous work, Falchikov (2005) discusses eight commonly used strategies for differentiating group and individual marks:
multiplying the group mark by an individual weighting factor;
distribution of a pool of marks;
separation of product and process;
equally sharing a mark but with exceptional tutor intervention;
splitting group tasks and individual tasks;
issuing yellow (warning) and red (zero grade) cards to individuals perceived as not ‘pulling their weight’; and
calculating individual grades in terms of deviations from the norm.
Peer assessment, where each member of the group ‘rates’ contributions of other members, has also been used to address the issue of equity but concerns are often expressed about assessing individual contributions fairly (James, McInnes & Devlin, 2002). The strategy of using peer assessment summatively remains unreliable despite procedures designed to reduce bias (Kennedy, 2005; Li, 2001). Engaging students in discussion about the selected strategies provides an avenue to enhance peer assessment of group work with potential benefits for reliability and validity.
Linking formative and summative assessment
Students are often anxious about assessment, especially in the online environment when they are unfamiliar with the technology. In a totally online environment they may feel inhibited in asking for clarification. They may need more support than you are accustomed to providing but early, effective support through formative or low-stakes assessment can assist students in preparing for major summative assessment tasks.
It is desirable for learning if you design summative assessment tasks which also have a formative aspect so that you provide students with feedback, as well as a grade. In doing this, it is important that you make the most of these formative opportunities to guide learning and offer support, while providing a rationale for the grade.
For example …
Consider the following points for structuring individual student comments.
Begin with a positive, supportive statement. Even if the student has not successfully addressed a topic, it is possible to make a comment which establishes rapport, and emphasises the support available to deal with difficulties.
Follow this with well organised, specific information on aspects which have been dealt with well, and constructive advice for addressing problem areas.
Try to end with a positive, forward-looking statement which reinforces avenues for further help, if necessary.
As we discussed in Chapter 3, remember to offer feedback to students who are performing well, not only to those who need help. The above example combines informative feedback with acknowledgement feedback. The latter without the former type of feedback is of limited value for learning.
Scheduling assessment tasks carefully
In linking formative and summative assessment, scheduling plays an important part in assessment design. It has increased impact in a fully online environment when communication around assessment tasks may be the main form of contact you have with your students. Establish regular contact related to assessment beginning with an early diagnostic task.
For example …
Set an early diagnostic assessment task to make contact with students, determine their needs, and provide an opportunity for early guidance and feedback. This will help to develop students’ confidence, establish a relationship with them, and provide some early direction for future assignments. If you have a large enrolment, devise a task which does not require a lengthy comment, and to which you can respond on a simple marking form designed to fulfil the role of providing feedback. Speed up your feedback by responding online.
Schedule assignments throughout the semester so that there are opportunities for progressive feedback and ensure that you provide feedback well before the next assignment is due.
Where possible, try and ensure that the tasks you set do not clash with assessment tasks for other, related subjects.
Although it is important that students receive early feedback, it is helpful if early assessment tasks have minimum summative weighting while the students are becoming familiar with requirements. Providing quality feedback while minimising assignment turnaround times can be a difficult balance to achieve (though we have already mentioned some options in relation to ‘who assesses?’ and more will emerge as we consider some of the online options).
Continuous assessment, through a set of assessment tasks (preferably interlinked) is much better for learning than many of the alternatives, and it is another strategy which reduces opportunities for plagiarism. It can also contribute to validity of authentic assessment tasks which are individualised for students. However, it does present some challenges in terms of management. It means continuous time spent assessing rather than the ‘one-hit’ of marking an assignment or examination. While continuous assessment is valuable, take care not to over-assess, as that will have negative consequences for both you and your students. All you need is an appropriate sample of student learning that reflects the unit objectives.
Using assessment criteria for grading assignments
Criterion-referenced assessment (where grades are allocated against assessment criteria which are determined by the learning objectives) provides more opportunities for improving learning than normative assessment (where grades are distributed according to a normal curve). It is helpful to both markers and students if assessment criteria are included on a marking form. You need to decide whether to design a rubric (grading scheme) for this purpose. Assessment rubrics can be very useful for summative assessment as they help describe what is expected and how each element (objective) will be rated. However, in some cases they can ‘normalise’ grades, and inhibit creativity. Students may consider the rubric as a list of boxes to be ticked rather than treating the assessment task as an opportunity for learning (Panko, 2006). An example of a rubric is provided in Figure 4.1.
If there is potential to negotiate assessment criteria for specific assignments with students within the scope of the objectives, this can provide a further assessment role they can undertake to benefit their learning.
Engaging students in the assessment process can be a motivating strategy as well as a learning opportunity for them. As they learn more about assessment as a process there is the potential for them to develop self assessment skills which are critical for lifelong learning.
Considering reliability and validity
For summative assessment purposes, you need to design tasks so that assessment is:
reliable – marking is consistent each time the task is administered; and
valid – it gives you the best evidence possible of the abilities being examined.
You may need to implement measures to ensure that items can be consistently and objectively assessed, especially if there are multiple markers. Attention to assessment criteria and rubrics should help you to meet this requirement. It is also important to ensure that appropriate weighting, which reflects the specified objectives, is given to components within assessment tasks and across the range of assessment tasks that you provide.
A further issue to consider is that traditional notions of reliability and validity are challenged by recent developments in thinking about assessment that support authenticity and emphasise the highly contextualised nature of learning involved in preparing students for life and work. Boud and Falchikov (2006) comment on the socially constructed, highly situated nature of learning in these settings, noting that this often involves collaboration in teams. In these circumstances, the assessment criteria must be determined in each situation. This creates a tension as it highlights the intrinsically local nature of assessment and means that judgements of general achievement based on local practices are not reliable and valid (Knight, 2006). It suggests that helping students to represent their achievements to employers requires differentiated approaches to assessment so that reliable and valid approaches are offered alongside local, contexualised assessment (Knight & Yorke, 2004).
The online environment offers excellent opportunities for contextualised and differentiated approaches to assessment (such as debates, case studies and role plays), while the inclusion of online peer assessment in communities of practice or communities of inquiry provides valuable dimensions to team work, preparing students for peer review in workplace situations as an aspect of local, contextualised assessment of complex levels of achievement. Developments in games, simulations and virtual worlds represent another dimension of contextualised learning experiences where assessment for and of learning merge, with evidence of reliability and validity transparently identified by the extent to which challenges are met. As we noted in Chapter 2, the related merging of real and virtual worlds potentially offers major benefits for assessment in life and work settings where simulated experiences offer immediate evidence of a student’s ability in related ‘real world’ contexts.
Communicating the assessment task
Although this may seem like stating the obvious, it is important to ensure that you communicate your online assessment requirements clearly and completely to students. This includes details of the task itself and how you will grade it, as well as scheduling and submission arrangements. This is particularly important for summative assessment in a fully online teaching environment where your opportunities for ensuring that students understand the requirements are more limited. To avoid misinterpretation by students (or repeated requests for clarification), try to think of all the questions students are likely to ask about a topic and convert your answers to a piece of one-way communication. Be especially careful in setting out details when complex processes are involved (including group work): multiple communication channels may be necessary. At the same time, design your instructions carefully (perhaps by chunking them appropriately) so students do not have to wade through a large volume of information in order to work out what they need to do!
For example …
You will conduct a debate on a given topic. You will be assigned to a syndicate either on the affirmative or negative side. Details of your syndicate, the topics, resources and guidelines are given on the LMS site for this subject. You will research your topic and use the allocated online discussion space to prepare your arguments, including preparing for:
The debate activity is a group assignment which contributes up to 15 per cent of your final mark: 5 per cent for preparation and 10 per cent for delivery of the debate.
Your work will be assessed from your discussion posts and the debate itself according to the guidelines for conducting the debate given on the LMS site.
For first year students, detailed information on expectations relating to literacy, content and referencing (and also cheating and plagiarism) is particularly important. Even when your purpose for assessment is diagnostic you need to tell students why you are assessing them, for what (what aspect you are trying to diagnose), and what you will do with the information (if you diagnose a difficulty, will you help students resolve it?). Sometimes the provision of model answers can be a great help to students.
In summative assessment you will need to articulate the criteria by which students will be judged to have been successful in meeting each objective. This is not necessarily a one-to-one relationship; you may have several criteria addressing one objective and/or one criterion which respond to aspects of several objectives. Both the objectives and the assessment criteria should be made clear to students.
For example …
In the debate example above you may also assess students individually on the basis of a report on the debate itself where they discuss both sides of the argument and arrive at a conclusion. Including objectives and assessment criteria in the instructions ensures that students know what is required of them and how they will be assessed.
In this example Objective 1 is addressed in criteria referring to ‘accuracy and relevance of information’ and ‘breadth and depth of content’; and Objectives 2 and 3 are addressed in criteria referring to scientific thinking and problem solving; Objective 4 is addressed in all parts of the task.
This section has illustrated issues related to assessment that are generally applicable. Now we will consider particular uses of the online environment that can benefit assessment.
Using the online environment for assessment
Uses of the online environment that are available for both formative and summative assessment fall into the four main categories indicated in Table 4.2. Each use offers opportunities for reconceptualising assessment to increase authenticity but usually these opportunities will be greater in relation to the last two (online discussion and web publishing).
We will now address each of these uses, discussing issues that they raise and summarising opportunities and challenges that you may need to consider. Bear in mind your learning objectives, your students’ characteristics and your learning and teaching context, to identify the points that are relevant to you. The opportunities can be pedagogical, managerial or administrative, and the benefits can be to students, teachers or both. It is helpful to identify the nature of the benefits you expect so you can offset them against challenges.
Use 1: Submission of items
Assignment submission is the most frequent use of the online environment for assessment (mainly summative assessment). It is the simplest form of online assessment to implement (though the administrative aspect of compiling grades online can be more complex).
Online submission is most likely to be used when an assessment item is a piece of work prepared by an individual student, for example, an essay. It facilitates the management of the assessment process by simplifying submission and return of assignments and keeping a record of this process (including dates and times). You can provide an assignment cover sheet for students to download, complete and submit as the first page of their assignment. The cover sheet may have been developed for paper copy completion so you will need to guide students to complete the declaration by typing their name rather than submitting a signature. Alternatively, you can use the online environment to streamline this process.
File formats
The item submitted is commonly a Word document, but can be any electronic file format that the assessor can access. Other file formats increase the possibilities for reconceptualising a task to take advantage of the en-vironment. The file format will depend on the nature of the assessment item: for example, an essay in Word format; graphics in jpg, gif, png or any of the many other possible image formats; an interview as an audio or video file (mp3, wav, aiff, etc); data in spreadsheet format (Excel); an animation as a multimedia file (Flash, exe, etc.); a webpage or website (html); or a presentation might be submitted as a PowerPoint file. In the last case, consider adapting the file format to one which suits the medium. A PowerPoint file, which is designed to support a presentation, even when annotated, may not be the most efficient format in terms of preparation (for the student) and marking (for the assessor). For instance, the same assessment item could be submitted as a Word document or html file. Choosing the best file format for an assessment item is an important aspect of online assessment design.
Giving students options in regard to the file format may allow them wider choice in the way they express themselves and expand opportunities for creativity. The key point here is that attention to something as simple as a file format can provide an easy way of making an assessment task more authentic. In this case an existing task may be transposed to the online environment, but the way students present their responses is reconceptualised. You will need to consider equivalence in assessing alternate file formats, ensuring that students are able to meet the criteria whatever format they use, and that those who do not have access to some technologies which may enhance presentation are not penalised.
A submitted assignment can be downloaded, assessed and returned electronically to students with comments. Where an assignment is in Word format, feedback to students can be given as ‘tracked changes’ on the file itself, or in a separate file (for example, Word, Excel). Pdf is a secure format for giving feedback to students. A student submission converted into pdf format can be annotated, have a marking sheet or other file attached, and be returned electronically.
A practical issue to consider in designing the assessment item is the likely size of submitted files. This has implications for the time taken and bandwidth required for students to upload, and implications in terms of class size.
Successive submissions
Assignment components can be submitted successively, ‘building’ an assessment item in stages, and progressing from formative to summative assessment. As mentioned previously, assessing in this way can help address plagiarism concerns as progress can be monitored.
For example …
Early in a semester students might submit an essay plan on which you can give formative feedback. They could then develop the plan into a final submission, either directly or in stages.
For an early assessment task, you might also use a submission tool diagnostically. If you wish to know something about the experience of your students, you could require them to submit an example of previous work or a summary of past experience – or ask them to undertake one of the diagnostic tasks mentioned previously, such as writing a letter about themselves, or commenting on their goals for the unit. This could then be related to subsequent formative and summative items. Of course, if you want everyone to share their experiences then you would use a discussion space.
Table 4.3 summarises some opportunities and challenges relating to online submission of assignments.
Use 2: Automated assessment
Assessment items which are automatically delivered to students, then scored and returned to them with automatic feedback, can be effective and efficient for both teachers and learners. This use illustrates interaction with the computer interface as a form of activity that is possible in the online environment which we did not discuss in Chapter 3 because it is usually implemented for formative or summative assessment purposes. The most commonly used form of automatic assessment is the quiz, but interactive multimedia programs can also offer valuable learning and assessment opportunities. Whatever your use of quizzes, the tracking potential of the online environment can offer valuable insights into your students’ learning. This may help you to identify common misunderstandings and misconceptions easily, allowing you to focus student learning more effectively.
Quizzes for formative assessment
In Chapter 2 we discussed the use of an online quiz to provide automated feedback for learning. In the example we used (Figure 2.3) feedback was given for each of the multiple choices offered. Quizzes also provide effective opportunities for self assessment and revision for students. Just as text books often have a quiz at the end of a chapter, an online quiz at the end of a topic can give students a measure of how well they have addressed the chapter or topic. A significant benefit of online quizzes is that feedback is immediate. If feedback is not incorporated into self assessment quizzes, valuable opportunities for learning and motivating are missed. Feedback in quizzes is more complex for short answer and paragraph type questions than for other quiz formats, but model answers can provide points of comparison and can be programmed in advance to give students immediate feedback.
Quizzes for summative assessment
If you are using online quizzes for summative assessment you need to consider verification and invigilation as well as technical issues relating to numbers of students concurrently accessing material. Conducting the test in a computer laboratory is the easiest way to provide for security and also technical troubleshooting. However, this removes one of the major advantages of online quizzes – the ability to provide some flexibility in the time and place of testing. Strategies for summative testing which address verification to some extent include:
scheduling the test to be undertaken within a limited time from commencement (while making it available over a broader timeframe);
permitting students to see only one question at a time and not allowing each question to be revisited;
including some short answer questions which relate to other authentic assessment items they have completed; and
using question banks which are large enough to randomise equivalent questions within different sections of the test.
A significant advantage of using quizzes in the online environment is the potential for students to see high quality images (see Figure 4.2). However, when images are integral to a quiz for summative assessment you must consider the time taken to download them.
Figure 4.2 An image-based quiz question*
* This image appears online at a high resolution showing detail not evident in the print version.
Multimedia options
Multiple levels of assessment can be built into interactive multimedia applications which are included in LMS sites. The cost of production may be warranted if the application addresses a learning problem that has been difficult to address in other ways and they offer more flexibility in design than LMS tools.
For example …
A molecular mechanism is difficult to describe in words and static pictures do not indicate how molecules interact. In this case an animation of the mechanism can be helpful, and students can be guided to make decisions (for instance, via a drag-and-drop activity) and receive feedback depending on their decision.
In Chapter 2 we showed an example of a multimedia version of a multiple choice question (Figure 2.4). This was one question in a large programme of activities that students complete to explore DNA structure. In this particular question students identify the nature of the bonds between DNA strands. The consequences of their decision are ‘played out’ in animations to explore why they have chosen correctly (or not). Students receive the feedback automatically as soon as they make their selection.
These types of activities provide good opportunities for diagnostic and self assessment, and the automatic aspect makes them useful for formative assessment. They are only optimal for summative assessment if they have a built in scoring and/or auditing facility which allows student tracking and grading.
Table 4.4 summarises some of the opportunities and challenges associated with automated assessment.
Use 3: Online discussions
The options that we have considered for online assessment so far have addressed assessment of items and feedback to individuals. Now we will look at online opportunities for communication and collaboration that can involve individual, peer or group assessment. Online discussions are easily and quickly set up and provide excellent opportunities for formative and summative assessment.
Formative assessment
A major benefit of the use of online discussions for formative assessment is that they allow prompt feedback to students on common queries and difficulties, or on set activities. An administrative advantage to you is that you only have to provide the feedback once. Peer assessment often occurs when students respond to queries from others. You can encourage this generally (through positive reinforcement) or more specifically by asking individual students to respond to a query or issue.
Summative assessment
When learning objectives include verbal communication, collaboration or project management, summative assessment via online discussion is a good option. In defining your expectations consider the design of the assessed online discussion task, as well as how you will grade the discussion.
Designing an assessed discussion task: Discussions may be used to help students complete a task individually or as part of a group. The discussion itself may represent the assessment item or the discussion may contribute to the development of an assessment item.
You will need to consider questions such as: Will student participation be enough or will you assess the content of students’ posts? How many posts will students need to make? Will the quality of their writing matter? Will they need to cover particular content? Will students need to respond to each other or just post for all to read? Will peer commentary be important? If students need to engage with each other, will they do that on a one-to-one basis or on a one-to-many basis?
If an objective of assessing a discussion is related to students’ project management skills then it might be important to stay out of the discussion. If you are assessing the content of their posts and students are straying off the track, then your contribution to put the discussion back on track may be invaluable for learning as well as for the assessment. By giving formative feedback during the discussion you can assure students that they are addressing the objectives appropriately.
As for all assessment items, students need to know the timeframe of the discussion. For students in early years of their studies and those unfamiliar with online discussions, specifying a timeline for each post as well the final post or due date can help relieve their anxiety. As we mentioned in Chapter 3, it is often important to provide specific dates: one for when an initial posting is required; and another for commentary on initial postings. Otherwise, students posting late contributions can place unfair pressure on other students by not allowing them sufficient time to make a required response.
Where a discussion is used for assessment early in the semester, students often need to be given considerable support to ensure that they have solved access and technical issues and are comfortable with the online environment. You should avoid setting complex discussion tasks too early. Use of Salmon’s (2003) model for teaching and learning online can help you to develop tasks of increasing complexity to meet higher level learning objectives as the semester progresses.
For example …
At the information exchange stage of Salmon’s model (Stage 3) you might ask students to search for evidence about the social impact of natural disasters and make a brief posting providing a resource they found and explaining why they selected it.
Later, at Stage 5, you might ask them to analyse the resources provided by the class (perhaps, along with other information that you provide), to prepare a final piece of work on this topic that includes recommendations for action at some level.
Although innovative assessable activities involve reconceptualisation, you can also simply transpose activities that you might run in a classroom to the online mode with the advantage that a record of discussion and the inputs of individuals is created. Further advantages for students include flexibility in relation to time and place, and the ability to take time to research topics and craft posts. This can be a particular advantage for shy students and those for whom English is a second language. As explained in Chapter 3, you also have the option of scheduling students as e-moderators. Their performance of this role could be part of your assessment of them (if it relates to the objectives) or it could be a valuable opportunity for them to undertake the role of assessor.
Scoring contributions to discussion: The mark allocated to a discussion as a proportion of the overall mark for the unit should be consistent with the objectives that the discussion addresses. Your answers to the questions we asked about designing an assessed discussion task will have an impact on scoring.
Marks for the quality of writing might be relevant where objectives include writing skills, or where discussion contributions form part of a larger assessment item.
You need to consider how you will differentiate and award group and individual marks. In considering your expectations of students you are defining criteria for assessment.
Assessing group work via discussions: Online discussions provide an excellent avenue for assessing group work because of the record of individual contributions. This helps address the issue of student dissatisfaction with group work mentioned earlier. Student negotiations around working as a team can be used as evidence of collaborative skills, and contributions of content and ideas can be attributed to individuals. The discussion may be the sole output for assessment or may be used to assess collaborative and individual components where the output, which is also assessed, is external to the discussion. The external item might be a document such as a report, a website, a video or audio, a presentation or even a performance.
Table 4.5 summarises some of the main opportunities and challenges associated with assessing online discussions.
Use 4: Web publishing
Benefits of web publishing for assessment include the potential for authenticity and the motivational value of preparing work to be published which then contributes to improving the quality of the learning outcomes.
A published outcome may also highlight the benefits of working in groups on collaborative projects. The opportunity to compare and critique the published work provides further avenues for extending the quality of the learning outcomes. We use the term ‘web publishing’ to include contexts where the outcome is available to the world (open access on the internet), or a subset of viewers which may range from a large number (e.g., everyone in your institution) to just two people, the student and yourself. Depending on the nature of the task, you may need to provide students with guidance on writing for the web, including writing style, usability (including linking) and visual layout. Informal writing may be suitable for a personal journal or microblogging, whereas a webpage may require a specific style and format.
In this section we cover again common ways in which students might create and publish content on the web that we looked at in Chapter 3, in order to make some comments on the use for assessment of:
We do not list mobile technologies separately in this section because their use can contribute to all of the above, in capturing data that is used in published content, or in its dissemination. Dissemination could involve open publication on the internet (e.g. videos uploaded to YouTube). Data captured by mobile technologies may also be included in the submission of items for assessment and could be related to discussion items that are assessed (Uses 1 and 3).
E-portfolios
If you are using e-portfolios in your learning design you will be able to configure the software to streamline assessment. Students will be able to collect relevant evidence required for assessment and integrate the feedback they receive. It is important that the control of the e-portfolio ‘publishing’ remains in the hands of the student to derive maximum benefit from both learning processes and the products to be published. The student will prepare the ‘published’ view by selecting the required evidence and annotating it appropriately.
Blogs
To facilitate students’ reflective practice you may have chosen to use blogs. Like an e-portfolio, this is predominantly an individual activity but one in which the technology allows for immediate and targeted feedback to the author (student) by others. In terms of assessment, a key aspect of a blog is its chronological nature which provides evidence of development over time.
This is suited to a continuous assessment strategy. While you may restrict access to the blog if it is an assessment item, in some cases (depending on your objectives) an open blog will contribute to its authenticity and you might include consideration of appropriate comments by members of the public in assessing the extent to which the blog meets its purpose.
Similarly, while you may think microblogging an unlikely option for assessment, it could provide useful evidence of the ability to present key ideas briefly, with responses by members of the public validating the extent to which it meets its purpose.
Wikis
In contrast to e-portfolios and blogs, wikis are suited to co-creation of content by groups of users and to assessment of group processes and products. In Figure 2.6 we showed the homepage of the output of one student group in a project which was undertaken using a wiki. As we have mentioned, a significant problem in assessing group work has been student dissatisfaction when contributions of group members are uneven. The visibility of all contributions and the process of the group work recorded in the history of a wiki effectively address this aspect of assessing group work.
Shared documents
If a group assessment task is to create a Word document, web-based applications such as Google Documents may be a good option. Students work on preparing the file and can export it for submission or inclusion in a larger project.
Student podcasting
If students have engaged in podcasting as a learning activity you may simply assess the audio or video product of the activity. A reason for choosing podcasting as an activity may have been ease of access for students and this will be a benefit for you as an assessor also. However, you could also consider the skills that students would have developed in the process of creating the podcast, including technical aspects as well as those related to syndication, such as choosing appropriate metadata.
Table 4.6 summarises some of the opportunities and challenges associated with the use of web publishing for assessment.
Supporting students
We stated at the beginning of the chapter that our assumption was that online assessment should either offer a benefit to student learning or, at least, should not inhibit it. We also suggested that considering your learners’ characteristics and the context of their learning was an important aspect of assessment design. In addition to the specific knowledge that you have about your students, you should check that you have addressed the following general issues.
Have you considered student support when you designed your assessment tasks, particularly if the technology or the assessment approaches are new to them? Have you established technical support procedures and conveyed them clearly to your students?
In considering the above questions, have you allowed for diversity among your students, including issues such as access and equity, variations in competence with the technology, the needs of international students, and any costs or ethical issues?
Have you discussed with students their responsibilities and your expectations of them, relating to issues such as: participation; self, peer or group assessment; authentication; and specific arrangements relating to online assessment tasks? This requires you to think about the way you communicate with your students and might involve consideration of learning contracts or other approaches to demonstrate student involvement in and commitment to the approaches that are being implemented.
Above all, are you approachable and is this evident to your students? Online assessment, particularly when it is not undertaken in a computer laboratory and is used summatively, can result in high levels of anxiety for many students, and given the potential for high transactional distance in these situations, it is extremely important that students know what to do, and that they will receive a sympathetic response and practical assistance, should problems arise.
Your ability to support your students may depend to a large extent on your ability to support yourself and to manage the situation. This ranges from managing broad technical, security and integrity issues to personal issues relating to your own use of the online environment for assessment, and managing the particular characteristics of your students. In the following section we will consider some of the broad issues relating to management of the environment first and then move to some of the specific management and administrative skills you may need to have.
Managing and administering assessment
Technical aspects
You would not be considering online assessment unless you were sure that both you and your students had access to appropriate hardware and software, and also appropriate bandwidth if students will be accessing their assessment tasks outside the university. It is worthwhile conducting an audit to check your students’ levels of access before you proceed with your planning, particularly if you are considering the use of timed summative assessment tasks. While you may be reasonably sure of the security of an LMS platform, there is always the possibility of technical problems when technology is used so you should have a backup plan in case of problems.
Some specific issues to consider include the following.
Ensure that your students can authenticate their access to the online environment in which you are offering assessment, and that they know where they can receive support on such issues.
Will students need to download large files as part of their assessment? This is especially important for tasks conducted within limited timeframes.
Have you taken into account scheduled platform maintenance periods? If you are planning a major online assessment activity where timing is critical, contact your platform administrator to help you to maximise the possibility that no unscheduled disruptions affect your assessment.
If you are using a system that is not institutionally supported (for example, blogs or wikis that contain assessable work), then it is extremely important that you take responsibility for investigating access and security issues and arranging backup procedures with and for your students.
When things go wrong there is often a tendency by students (and staff!) to ‘blame the technology’. A cautious approach to technical issues will prevent many potential problems from occurring but, as indicated above, you should always have a backup plan. The second part of this equation is to make sure that students know what to do if technical problems arise. Make sure you make the most of the technical support available within your institution. If there is a helpdesk make sure you know how to access it, when it is active and the type of support it offers. Direct your students to this support rather than trying to solve technical problems yourself. Your time is better spent supporting them in their learning.
Authentication
Authentication issues are often at the heart of staff reservations about the use of online assessment: ‘How will I know whether the student or someone else completed the task?’ Of course, the same question can be asked about almost all assignments that students submit which are not written in an examination room. This issue can be addressed, in part, through attention to assessment design. We have already mentioned design strategies which help to address authentication concerns but we summarise some of them again below.
It will be more difficult for someone else to impersonate the student if the assessment items occur in a tightly structured sequence, where each successive component builds upon the previous one and/or where these require analysis of the student’s own inquiry and experience, possibly related to the development of portfolios of work. Authentic assessment items, by their nature, tend to work against plagiarism.
The online environment is extremely beneficial in keeping a record of the process in the development of an assignment, as well as providing a variety of ways for presenting the final product. Assessment tasks where you can see development taking place in the online environment, or those that include commentary on products as they develop, provide useful ways of monitoring authentication.
Randomisation of questions and careful timing helps to limit collusion between students when quizzes are used, with completion in computer laboratories an option for on-campus students.
The issues of cheating and plagiarism are of huge concern, irrespective of whether assessment is conducted online, although, of course, it is access to the internet that has expanded the ease with which students can obtain and misuse information. You should be familiar with your institution’s definitions relating to cheating, plagiarism and collusion. We noted earlier the inclusion of assignment cover sheets with assessment submissions or an equivalent online declaration. These allow you to meet institutional requirements regarding use and inclusion of plagiarism information for non-examination assessment.
It is important that students can identify plagiarism and issues of attribution, so provide information and links to resources within your institution or beyond. Alternatively, you may go further and design an activity which involves identification of plagiarism in content related to subject matter that your students are studying.
This has potential for richer learning outcomes. While the use of plagiarism software can be viewed as a deterrent, it is also a learning opportunity for students who are not always aware of the nature of plagiarism. Keep in mind the limitations of the software and ensure you discuss its implications with students. This again relates to student support, helping them (especially new students) to understand what is expected of them so that penalties are clearly a matter of last resort.
Your administrative skills
While online assessment offers a number of administrative benefits, your ability to access these will depend on developing your own skills in relation to the tasks involved. Below we cover some of the tasks you may need to undertake.
Setting up your site
Administrative tasks involved in setting up your site for assessment include organising the assessment tools required for assignment submission, discussions, creating quizzes or web publishing. If you are using any aspects of group assessment you will need to be able to divide students into groups. If discussion topics are used for individual or group activity, you might also want to think about naming conventions for messages relating to specific assessment components, as well as providing information about netiquette and advice relating to the length, timing and other requirements for specific postings. There may also be a range of additional tools or resources that you need to add to the site, depending on the nature of the assessment.
Managing grades
If you are using an LMS, you are likely to have access to an administrative tool for managing grades. This is a spreadsheet within the system for storing student records. Grades are automatically created for automated assessment tasks with pre-determined answers, such as multiple choice tests. Assignments that you manually grade within the system can also be automatically added to the appropriate column. You can hide or reorder columns, and include columns for assessment that are undertaken outside the system. You can include both alphabetical and numerical information and can release grades to students via the tool available for this. You can create columns to calculate end of semester/term grades and you can download the file to your computer and open this as a spreadsheet. Once you are able to manage this efficiently it should save you a lot of time in managing assessment and provide a concurrent benefit to students by speeding up their receipt of feedback.
Managing workload
A major concern that is frequently raised by teaching staff about teaching or assessing online is the workload involved. In relation to assessment, particular concerns emerge if you are implementing continuous assessment through related activities, or assessing online discussion which requires ongoing e-moderation. Once again, we would urge you to consider carefully your assessment design, in conjunction with the question of ‘who assesses?’ in order to identify strategies that are manageable while also contributing to the quality of learning.
Managing risks
Having considered the uses of the online environment for assessment and some of the opportunities and challenges, together with a range of support, management and administrative issues, you will now be in a good position to evaluate the risks associated with implementing online assessment. Table 4.7 provides an example of a simple way of doing this. Rating risk factors as high, medium or low will allow you to assess whether a risk is acceptable or not in your context and what you might do about it.
With rapid evolution of technologies the nature and level of risks change so it is important to review them periodically.
Illustrating the ideas in this chapter …
Suzy continues her story below to illustrate how she addressed aspects of online assessment.
Summary
You should now have clearly in mind the relationship between assessment and student learning in order to consider this in designing assessment for the online environment. In identifying reasons for assessing your students, we have distinguished formative and summative assessment and particularly noted the role of formative assessment in enhancing learning. This role has been evident among the assessment design principles we have considered, as has the role of authentic assessment. In preparing for online assessment we have also thought about the potential of self, peer and group assessment and recognised that online assessment may have learning benefits or administrative benefits (or both). We hope that considering assessment in the online environment has helped you to recognise ways to address a wider range or learning outcomes. You should now have a good framework for considering in more detail the way you might use the online environment for assessment, and some of the potential opportunities and challenges.
We continue building the checklist by adding issues related to assessment online. As we mentioned in previous chapters you should feel confident of your grasp of the issues if you can answer ‘Yes’ to a majority of the items in the list. If that is not the case then you may need to look to colleagues who have been engaged with assessing online and learn from their stories.
