Fernando F. Padró

Introduction

St John, Kline and Asker (2001) wrote about the need to rethink the links between the funding of public higher education with accountability measures based on student-choice processes and student outcomes. Student outcomes are now considered the principal mechanism for determining successful performance of higher education institutions (HEIs). No longer are external reviews merely interested in the traditional input data; the focus is for input and throughput mechanisms to enhance and maximise student learning opportunities within the campus for traditional as well as the increasing number of non-traditional students. Description of programmes is not enough – universities have to know more about their product (Cronbach, 2000).

Student-choice processes place a greater emphasis on student-based information to identify needs in order to figure out how to better serve those needs. Student feedback is desirable because it helps determine student satisfaction with their interaction with the different elements of the university and measure the extent of active engagement by the students in terms of curricular and co-curricular programmes. The hope is that institutions will: (a) retain their students until graduation, (b) generate student learning that is attractive to potential employers or graduate programmes, and (c) enhance student loyalty as alumni to provide the university with funds, participative support, and feedback that allows the institution to provide a value-added experience while on campus and afterwards.

Traditional student information included in the application package (grades, entrance exams, personal information) provides only information specific to acceptance of students to HEIs and subsequent placement into residence halls, support programmes and/or courses (remedial, honours or advanced level). Grades alone are also insufficient to fully gauge student contentment with what they are getting out of their university experience. Thus, there is a need to know more about students and their satisfaction with their experience. From 1966 onward, the Cooperative Institutional Research Programme (CIRP), a national longitudinal study of the higher education system in the USA, has been regarded as a key source of comprehensive data on incoming and continuing students. An additional layer of student data has become available from 2000 onward through the National Survey of Student Engagement (NSSE), which annually documents on a national basis student participation in programmes and activities HEIs make available for learning and personal development. According to Kuh (2001), NSSE allows for the creation of national benchmarks of good practice for universities to use to measure their improvement efforts relative to academic concerns along with supportive campus environments.

The point of student feedback is to systematically use what had been collected informally for many years, such as information from alumni (cf. Braskamp and Ory, 1994), and formalise it so as to make it a meaningful component of institutional decision-making. This systematisation of data collection and the focus on incoming, current and former students (hopefully graduates) is shaped by the current reality of the nascent environment that Slaughter and Leslie (1997) called academic capitalism, an environment in which academic and professional staff have to navigate in more highly competitive situations driven by market-like behaviours.

Types of student feedback can be broken down into distinct categories (Table 2.1). Taken together, the model that emerges is a feedback process that has a striking resemblance to Brocato and Potocki’s (1996) customer-based definition of student and quality of instruction in contrast to the more traditional notions of academic endeavour and quality:

Quality focuses on two process-related questions: ‘What is wanted?’ and ‘How do we do it?’ (Straker, 2001). The inclusion and significance in the use of student feedback is an example of how quality management and assessment are more widely accepted today (Kitagawa, 2003), even if there are ‘fundamental differences of view of the appropriate relationship that should be established between higher education institutions and their external evaluators’ (European Association for Quality Assurance in Higher Education, 2005: 11). This is why the Malcolm Baldrige National Quality Award Education Criterion 3.1(b) 2 asks educational institutions how they:

And although voluntary regional accrediting bodies in the USA have not fully adopted the Baldrige criteria as the blueprint for external reviews, one can see a more detailed tactic in the New England Association of Colleges and Universities (NEASC) (2006) standard 4.50:

Student feedback in the evaluation of academic programmes and instruction

Student evaluation of courses or units and of instruction has been used at HEIs for many years. As far back as 1949, Guthrie asserted that teaching is best judged by students as well as by colleagues. Thus, in the USA, by 1994, 98 per cent of respondents indicated a systematic student evaluation of classroom teaching is occurring at their campuses, with the other 2 per cent indicating that their institutions are considering it (Glassick et al., 1997). In Australia, the Course Experience Questionnaire was fully implemented in 1993 and continues to be used to this day for the purpose of allowing for a comparison of programmes among Australia’s universities. Meanwhile, especially under the re-engineering of higher education occurring in Europe under the Bologna Process, most European universities are instituting some sort of student evaluation of teaching – with the emphasis seeming to be in the form of student satisfaction (cf. Wiers-Jennsen et al. 2002).

The remainder of this chapter is dedicated to discussing relevant issues and concerns regarding student feedback within the framework of evaluating classroom instruction. To say that this is a controversial topic is an understatement. This is why Aleamoni’s (1999) literature review discusses and then rebuts many of the myths surrounding the use of student feedback as a means of evaluating instruction and instructor performance. What will come through is the challenge of framing the process and use of student feedback in relation to the job performance of instructors. A continuum seems to be developing. One side is represented by Arreola et al.’s (2003) meta-professional model for faculty, while on the other side is the service quality instrument – SERVQUAL-which others have made applicable to higher education first developed by Parasumaran et al. (1985, 1988). The rationale behind SERVQUAL is to measure consumers’ perceptions of quality when there is an absence of objective measures. This approach, as can be seen, easily aligns with many of the concepts driving the need for student feedback.

Case study

Seashoal Lows University (SLU) is a medium to large, urban comprehensive teaching-focused university of about 8,000 undergraduate and graduate students majoring in the arts and sciences and professional programmes in business, counseling psychology, education, and human and health services (a Master’s Large Comprehensive institution under the current Carnegie Classification Index). It has been using student evaluations of its instructors for a number of years. However, the faculty senate, the faculty collective bargaining unit, and individual faculty have been complaining over the appropriateness of how this is used and the purpose behind it. Student evaluations of faculty (SEFs) are externally created instruments given either through paper and pencil instruments or online. These are given near the end of the academic term. Problematic to the faculty and its related university organisations is that these instruments have become the primary element in deciding faculty promotion and tenure. According to the Agreement faculty have with SLU, the review process should be based on a portfolio provided by the instructor going up for review that includes publications, conference presentations, external and internal funding awards (when applicable), course syllabi and materials, classroom observations by peers selected by the applicant and the administration, observations by the supervising administrator, student evaluation of faculty results, documentation of university and community service, and external reviews of the portfolio by individuals agreed to by the faculty member and the head of the academic unit. Decisions are made by a committee within the School, with recommendations given to the Dean who, in turn, makes a recommendation to the Provost and President.

There are two principal types of faculty: faculty who have asked to be considered for their research as well as instruction; and those who want to be primarily considered for promotion and tenure based on their instruction (with criteria ostensibly following the suggestions put forth by Ernest Boyer (1990) regarding the scholarship of teaching). Programme-level accreditation at times provides guidelines for promotion and tenure for certain programmes while the university’s overall criteria are purposefully kept nebulous to avoid potential litigation from those denied promotion and/or tenure.

The issue is that in practice, the main point of evidence has become the student evaluation of the applicant when it comes to instruction. It has become apparent that committees and responsible administrators weigh student evaluations disproportionately when compared to course syllabi and materials or classroom observation by peers and administrators. Faculty complaints range from inappropriateness of the instrument, because it is not linked or validated to institutional norms of good teaching, to the belief that student feedback really represents a popularity contest because they may not be the best judge of content. Some also are concerned that the time when these are given ties student observations to their idea of what their grade should/will be and, moreover, there are complaints that there is no real instructional support for faculty because there is no formal capacity to assist faculty – especially junior faculty – in improving their instruction.

SLU’s administration does not want to change the student evaluation process because they are concerned that the time and cost taken to create and validate an in-house instrument may adversely impact accreditation-related data analysis and reporting. They also like the idea that they can use the external instrument in order to compare instructor ratings with other institutions. Finally, the administrators believe that the external instrument is validated and, as a result, the instruments do provide an accurate evaluation of faculty, more so than the other factors that can be influenced through personalities and politics.

SLU realistically does not use the data collected for benchmark analysis although it can. It does not use the data collected for continuous improvement purposes or to at least check for problems with instruction. However, the University’s administration is mindful of the change in the external regulatory environment that identifies with the student as a consumer and wants to maintain and enhance responsiveness to student needs and expectations. Therefore, there is an impasse within the institution. One side discounts the merit of student feedback and at best is resigned to have to live with it. The other side sees student feedback as a way of maintaining its regulatory compliance requirements and finds that its ability to quantify instruction provides a more compelling measure to make career decisions.

Issues

There are seven issues the scenario brings up:

These have to be considered as a whole rather than as separate components because these are all interrelated. Regardless of approach taken, whether it is from a more traditional professional development model or a customer service model, there are legal implications to consider along with fitness of purpose, morale and buy-in concerns from all university employees and students themselves. Employees are concerned about job security and fairness. At play are the strategies individuals use as part of their sense-making process to determine what is needed for them to succeed at the HEI (cf. Weick, 1995). Apart from these aspects, it is important not to forget that students pay attention to the extent their information is actually considered and followed-up on, which in turn impacts their future engagement.

Probably the most important consideration is to identify and clearly articulate the reasons for and use of student feedback on instruction. For example, the American Council on Education and the American Association of University Professors (2000) jointly agree that stated criteria for tenure apply in actual practice. The extent to which these instruments are used for hiring/firing and promotion purposes may differ throughout the world. In many countries, feedback instruments are strictly utilised to measure institutional performance of academic units. Nevertheless, if the data reflect weakness at the individual level, individuals can be identified to receive additional support. Anecdotally, some HEIs identify the bottom 10 per cent of performers to give them training to improve their instruction.

The evaluation of academic staff performance reflects the role they play at their particular campus. Traditionally, the roles are divided into the triad of research, instruction and service, often in this order of importance. The roles of academic staff at a particular institution ought to be clearly defined and understood. It helps to know the criteria or standards for continuity and promotion. Review mechanisms and feedback have to be linked to these criteria or standards. Techniques and instruments need to be reliable and show practical and/or statistical validity. Techniques and instruments have to be weighted as part of the codification of value process relative to specific performance (Arreola, 2007). Techniques and instruments have the challenge of needing to overcome, as much as possible and feasible, Birnbaum’s (1988) characteristics of an anarchic university (unclear goals, imprecise technology, fluid participation, and solutions looking for problems). And, in addition, there should be an institutional support element at the campus-wide or unit level such as a centre of teaching and learning (cf. Padró, 2010; Institutional Management in Higher Education, 2007) to ensure improvement either in remedial or enhancement modes.

Two questions of role and use of student feedback in the area of instruction emerge, the one area of the triad where they are in a position to provide useful observations. The first question is: What is the role of student evaluations of instruction? Formative – focusing on diagnostics and continuous improvement – or summative in scope leading toward staffing decisions? Pallett’s (2006) observations provide an answer:

The second question is: What is the best type of instrument for students to use in giving feedback about courses and instruction? There are three approaches. The first one is for an HEI to develop its own instrument. This requires careful construction based on the institutional values of teaching, followed by appropriate reliability and validity studies that can take a few years to complete. Arreola (2007) described a ten-phase process for creating these forms:

Braskamp and Ory (1994) classify common forms of assessment of academic staff into three categories:

This typology suggests HEIs carefully consider the options available under this action plan. Regardless of type, without paying attention to the reliability and validity part of the process, questions arise regarding how good the data are and their usefulness. In addition, if staffing considerations are proved to be significantly impacted by this instrument, then there may be tortuous litigation ahead.

The second approach to utilising student feedback instruments is to use commercially available or ‘off-the-shelf’ forms created and sold by a private firm. This is the track represented in the case study.

Arreola (2007) does indicate that it may be a good idea to consider adopting or adapting a professionally-developed form because many locally-generated student feedback forms may not possess the necessary psychometric qualities of reliability and validity. These instruments have already been tested for reliability and validity. In addition, the use of such instruments allows for national comparisons between comparable institutions using that particular survey. Comparative data can be used for improvement reasons, personnel grounds and marketing/rating data that students and parents can use in making choices on which university to attend, and documenting performance of programmes and units as part of benchmarking exercises. A key aspect to deciding on whether or which instrument to select is to determine which one provides the university data that are meaningful to it based on how the data will be utilised.

Critics of using these commercial instruments have numerous concerns, ranging from appropriateness of the concept of student feedback itself to alignment of items/questions to institutional normative references of instruction to how data are interpreted and subsequently used. Some of these concerns echo what detractors think about home-grown forms. Ramsden’s (1991) research using the literature review by Marsh (1987) and others makes the case for an association between student learning and student perception of teaching. Aleamoni (1999) takes a look at these characteristic arguments against the use of student feedback from the lens of studies looking at these issues. What Aleamoni found was that most of the contentions do not hold based on available findings. For example, he countered the complaint that ‘Students cannot make consistent judgments about the instructor because of their immaturity, lack of experience, and capriciousness’ with evidence dating as far back as 1924 indicating just the opposite. ‘The stability of student ratings from one year to the next resulted in substantial correlations in the range of 0.87 to 0.89’ (p. 153). Aleamoni also countered another typical complaint, that ‘Student rating forms are both unreliable and invalid’, by arguing that ‘ [W] ell-developed instruments and procedures for their administration can yield high internal consistency reliabilities … in the 0.90 range’ while the vast majority of studies regarding instrument validity indicated the ‘existence of moderate to high positive correlations’ (p. 155).

The third approach toward what student feedback instrument(s) to use may not be a choice at all, because institutions are to use or adapt a national instrument required by national protocols or a national quality assurance process (cf. Institutional Management in Higher Education, 2007). The example in mind is the course experience questionnaire utilised in Australia, an approach the popularity of which is expanding to other countries, reflecting Ewell’s (2002) observation that assessment and accountability have become entwined. Linking accountability with the assessment of learning means, at one level, a change in the conceptual paradigm of assessment from an evaluative stance to that of assuming active and collective responsibility for learning while, at another level, it suggests the evolving of a learning organisation. The basis for this thinking is using a national instrument as part of a quality teaching framework in which students play a formal role, as is evolving in Europe (ENQA, 2005) and elsewhere in countries establishing their own national quality assurance system. According to the OECD:

In reviewing the literature and practices surrounding the use of student feedback for quality assurance purposes in the area of instructional performance of individuals, units and whole HEIs, what becomes apparent is that there is a continuum developing. One extreme emphasises professional development based on carefully structured multiple processes with adequate support mechanisms that help individuals succeed in their roles within the triad of research, instruction, and service. The other extreme is a strict customer service model based on documenting performance in instruction for institutional success, utilising comparative analyses so that different stakeholders can use this decision for informed decision-making (e.g. personal choice of institution, national policy determination of success and alignment to national intellectual/human capital needs). Below is a discussion of the models that exemplify the two extremes of the continuum.

The meta-profession model of faculty

‘Faculty performance is complex and dynamic’ (Braskamp and Ory, 1994: 22). Its evaluation should reflect the complexity of faculty work framed within clearly communicated institutional goals and expectations. The meta-profession model is selected as a highly systematised approach toward the evaluation of faculty that reflects these points. What this perspective represents is a review process that maintains the traditional focus of evaluating academic staff using the broader brushstrokes of competencies that cannot be defined only in terms of customer satisfaction.

Arreola et al. (2003) extended Boyer’s (1990) effort at making the professoriate and others rethink the role of academic staff at universities. Currently, academic staff ‘must perform at a professional level in a variety of roles that require expertise and skills in areas that often extend beyond the faculty member’s specific area of scholarly expertise’ (Arreola et al., 2003: 1). Arreola (2007) has taken this model and developed a comprehensive evaluation system in which student feedback on instruction is one of numerous approaches to collecting data. Student ratings are useful under the right conditions which, according to Aleamoni (1978, as cited in Arreola, 2007), is when they are used as part of a personal consultation between the instructor and a faculty development resource person.

The model is a multi-dimensional one, beginning with base professional skills that newly hired academic staff have. These include content expertise, techniques for keeping current in the field, practice and/or clinical skills appropriate to the field, and research skills and techniques appropriate to the field. However, these skills are insufficient when discussing instructional duties and other activities academic staff typically perform (institutional service, including potential administrative duties, and research aligned to institutional expectations). Arreola’s (2007) approach toward evaluating performance that includes all of these additional layers of expertise has eight steps:

SERVQUAL

For the purposes of this chapter, SERVQUAL represents a line of thinking in which student feedback is primarily for the purpose of determining the quality of educational programmes from a student or other external stakeholder perspective. SERVQUAL looks at ten variables to assess service quality fit: tangibles, reliability, responsiveness, communication, credibility, security, competence, courtesy, understanding/knowing the customer and access.

SERVQUAL is not without its detractors due to applicability to different service industries because of a lack of completeness in measuring certain aspects of service quality (Chiu and Lin, 2004; Sureshchandar et al., 2001). Others have methodological (Smith, 1995) and empirical concerns (Van Dyke et al., 1997). Thus, Carr (2007) proposes SERVPERF (service through the lens of organizational fairness) as an alternative. However, Bayraktaroglu and Atrek (2010) find that SERVQUAL as well as another similar model, SERVPERF (service performance), can be used in measuring service quality in higher education services. And in their use of SERVQUAL, Emanuel and Adams (2006) found that the dimensions of reliability (the instructor’s ability to instruct the course dependably and accurately) and responsiveness are the most important dimensions of instructor service to students.

Using SERVQUAL assumes that universities are a service industry. For example, Schneider et al. (1994) conceptualised teaching as a service ‘in that (a) teaching processes and experiences are relatively intangible; (b) teaching is typically produced, delivered, and consumed simultaneously; and (c) teaching typically requires the presence of customers’ (p. 685). ‘SERVQUAL assumes that the difference between the customer’s expectations about a service and his or her perceptions of the service actually determines quality’ (Bayraktaroglu and Atrek, 2010: 47). According to Parasumaran et al. (1985), it is more difficult to understand and evaluate the impact of services because they are intangible, heterogeneous and inseparable. The model distinguishes between service quality and satisfaction. ‘[P]erceived service quality is a global judgment, or attitude, relating to the superiority of the service, whereas satisfaction is related to a specific transaction’ (Parasumaran et al., 1985: 16). What is at play is a ‘disconfirmation paradigm’ that suggests that, prior to an interaction, consumers form expectations about ensuing product/service experiences (Prugsamatz et al. 2007). Figure 2.1 below illustrates how the customer service model focuses on sources of complaints and how failure can be traced back to reduce similar negative incidents in the future.

Discussion

In 1974, Hartnett and Centra found that there tends to be a good deal of consensus between administrators, faculty (academic staff) and students about the academic environment found at a university. The principal exception found was in how students differed in their views about democratic governance, with lesser variation found regarding the institution’s concern for innovation and the overall extent of staff morale.

Glasser (1998), in his analysis of quality in education for education in the twenty-first century, argues that students are asked to perform tasks in order to be evaluated, creating reviewable work that, in turn, allows them to recognise quality in the classroom. ENQA (2005) guidelines for HEI quality assurance systems include, among other items, student satisfaction with programmes and teacher effectiveness.

The challenge is to ensure these are mutually compatible in terms of improving student learning while enhancing professional development opportunities rather than punitive personnel decision making. Aleamoni (1999) warned of the potential misuse of student ratings due to misinterpretation and misuse by administrators for punitive personnel decisions. This warning could be extended to the use of data for quality assurance purposes as well. Embedded within this challenge is the balance between academic freedom in contrast to the need for conformance. Burgan (2006) suggested that variety of instruction is informative to students and universities. Conversely, UNESCO defined student evaluation of teachers as the determination of the need for conformance ‘between student expectations and the actual teaching approaches of teachers’ (Vlasceanu et al., 2007: 93) Programmes such as the Tuning Project being performed in Europe and Latin America – whose purpose it is to establish a methodology for designing/redesigning, developing, implementing and evaluating different study programmes (González and Wagenaar, 2008) – are concerned with conformity leading to disengagement of academic staff. This is because their interests (and those of the disciplines they represent) would become marginalised when compared to policy steering and satisfaction interests.

‘New research shows that outstanding performance is the product of years of deliberate practice and coaching; not of any innate talent or skill’ (Ericsson et al., 2007: 115). Contrary to a potential reading of comments in this chapter, the models being put forward at the international level regarding the evaluation of instruction understand this and insist that adequate support for improving instruction from the professional development and enhancement of student learning perspective be provided. What the role of student feedback brings to the debate of institutional quality is where the emphasis lies; particularly in regard to institutional autonomy when it comes to disciplinary interests, freedom of expression or inquiry, and pedagogical matters. Another way of looking at this is the potential that the sharing of conclusions happens because there is a preference to convene around failures rather than successes (cf. Darling et al., 2005).

This chapter is written primarily from an ‘American’ perspective. Observations are based on how this perspective fares in relation to international practice. The author first became aware of the development of a continuum in the practice of evaluating instructors when listening and reading approaches taken by academic staff in Europe to establish their own student evaluation forms. Some of these individuals used the SERVQUAL model because of a lack of awareness of long-standing models and methods for evaluating instruction and student ratings.

Writers such as Wiers-Jennsen et al. (2002) document how academic staff and institutions in Europe have been looking for workable models of instruction evaluation. The continuum represents the divergence that exists based on demands placed on the institution and staff. Student feedback is not the problem. What is pressing is the role student feedback plays in the form of student ratings and evaluation of faculty and how the data are going to be used. There are other issues that are not discussed here because they are important topics in their own right (e.g. electronic forms and the ability for students to see the impact of their feedback). What is the priority for academic staff? Is it an education that leads to jobs or developing critical thinking skills and aesthetic and ethical abilities that lead to needed social skills (e.g. Institutional Management in Higher Education, 2007) or is it research? Arreola and his colleagues have developed a conceptual model and a process for the instruction of evaluation that reflects the nuance of academic work. Is a customer service model sufficiently sophisticated and does it adequately align student feedback to ensure that all aspects of academic work are properly documented and enacted upon in a manner beneficial to academic staff, students, HEIs and nations? These are the questions which still remain to be answered.

References