Wednesday, October 17, 2012

2011 Agbetsiafa/Aghimien article in Journal of Current Research in Global Business


This document shows similarities in wording of the following article by Douglas Agbetsiafa and Peter Aghimien and the wording of earlier published authors.

Agbetsiafa, Douglas K. & Aghimien, Peter (2011) “Assessing Teaching Effectiveness in College Level and Accounting Courses Using Student Rating of Teaching.” Journal of Current Research in Global Business, 14(21): 44-52.


Agbetsiafa and Aghimien are both professors at Indiana University South Bend (IUSB).  Aghimien has been known to say: "If it looks like a dogwalks like a dog and barks like a dog, then it's a dog."  With a salary of $114,681 Aghimien is one of the top 10 highest paid professors at IU South Bend.  Agbetsiafa’s salary is also among the highest at IU South Bend ($90,040). 

Note that the UCLA references appear on a website (http://www.oid.ucla.edu/publications/evalofinstruction/index.html), but they were originally also available as a PDF with a 2006 publication date: http://www.oid.ucla.edu/publications/evalofinstruction/evalguide

From Agbetsiafa & Aghimien article, page 44:

Colleges and universities use student evaluations to assess quality of instruction or other aspects of a course. The data generated by these instruments assist an instructor to improve instruction or a course (Worley and Casavant, 1995; Boice, 1990-91). Administrators and tenure and promotion committees often use the data to assist in making tenure and promotion decisions. Administrators may rely on the data for helping make annual performance and salary decisions. These data are also used to help provide evidence of teaching excellence when faculty is nominated for teaching awards. Given the ways that student evaluations data are used in colleges and universities, it is necessary that the data derived from these instruments are valid and serve as reliable measures of quality teaching and course development.

Compare this to:

Student evaluation of teaching (SET) instruments are commonly used in higher education to assess quality of instruction or other aspects of a course. The data generated by SETs can be used to assist an instructor to improve instruction or a course (Worley and Casavant, 1995; Boice, 1990-91). Administrators and tenure and promotion committees often use the data to make tenure and promotion decisions. Administrators may rely on SET data for helping make annual performance and salary decisions. SET data are also used to help provide evidence of teaching excellence when faculty are nominated for teaching awards. Given the ways that SET data are used in higher education, it is imperative that SETs be valid and reliable measures of quality teaching and course development.

From Agbetsiafa & Aghimien article, page 44:

There continues to be robust debate and discussion about the findings of this extensive body of research and what conclusions can be drawn about student evaluations of teaching and their use. According to Algozzine et al, 2004, student evaluation of teaching is a very complex and controversial issue with inconsistent research findings (p.l38), while Kulik, 2001 argues that some studies on Student Evaluation of Teaching (SET) are "conflicting, confusing, and inconclusive" (p.l 0). Nevertheless, Kulik agrees with other studies that show that these evaluations are reliable, and valid measures of teaching effectiveness (Centra, 2003; Marsh, 1987; Penny, 2003; Spoorens and Martelman, 2007).

Compare this to:

There continues to be robust debate and discussion about the findings of this extensive body of research and what conclusions can be drawn about SETs and their use. Algozzine et al (2004) state that “Student evaluation of teaching is a very complex and controversial issue with inconsistent research findings” (p.138) while Kulik (2001) states that some studies on SETs are “conflicting, confusing, and inconclusive” (p.10). Nevertheless, Kulik concludes by agreeing with other studies that claim to show that SETs are reliable, can be validly used as a measure of teaching effectiveness and are useful in improving teaching (Centra, 2003; Marsh, 1987; Penny, 2003; Spoorens and Martelman, 2006).
[page 2]

From Agbetsiafa & Aghimien article, pages 44-45:

Research shows that students tend to take teaching evaluations more seriously than faculty and institutional members commonly believe. Students are more willing to participate and offer meaningful feedback when they believe and can see that their input is being considered and incorporated by their instructors and the institution. In general, however, students do not perceive that their feedback is often used. Some studies show that students place most value on evaluations for formative purposes, but research also indicates that students believe their input should be considered for summative purposes. Students would like to see more specific items related to teaching effectiveness on student evaluation of teaching instruments (Sojka, Gupta, & Deeter-Schmelz, 2002; Chen & Hoshower, 2003).

Compare this to:

Research shows that students tend to take teaching evaluations more seriously than faculty and institutional members commonly believe.  Students are more willing to participate and offer meaningful feedback when they believe and can see that their input is being considered and incorporated by their instructors and the institution.  In general, however, students do not perceive that their feedback is being used.  Some studies show that students place most value on evaluations for formative purposes, but research also indicates that students believe their input should be considered for summative purposes.  Students would like to see more specific items related to teaching effectiveness on student evaluation of teaching instruments.  (Sojka & Deeter-Schmetz, 2002; Chen & Hoshower, 2003)

From Agbetsiafa & Aghimien article, page 45:

Research also shows that faculty often believes that students do not take evaluations seriously and that ratings encourage grade leniency. Nonetheless, most faculty members pay attention to student feedback. Moreover, when evaluations are used for formative purposes, instructors show a high degree of motivation to improve their teaching based on student input. Studies have emerged showing how institutions and individual faculty members have begun using evaluations, consultations, and portfolios to improve instruction qualitatively. When faculty are well informed about the purposes of evaluation, much of their anxiety dissipates and willingness to learn from student feedback increases (Sojka, Gupta, & Deeter-Schmelz, 2002; Hativa, 1995; Gallagher, 2000; Bain, 2004).

Compare this to:

Moreover, faculty often believe that students do not take evaluations seriously and that ratings encourage grade leniency.  Nonetheless, most faculty do pay attention to student feedback.  Further, when evaluations are used for formative purposes, instructors show a high degree of motivation to improve their teaching based on student input.  Studies have emerged showing how institutions and individual faculty members have begun using evaluations, consultations, and portfolios to qualitatively improve instruction.  When faculty are well informed about the purposes of evaluation, much of their anxiety dissipates and willingness to learn from student feedback increases.  (Sojka, Gupta, & Deeter-Schmetz, 2002; Hativa, 1995; Gallagher, 2000; Bain, 2004)

From Agbetsiafa & Aghimien article, page 45:

Teaching evaluations are commonly considered for summative purposes, including tenure, merit increase, retention for non-tenured faculty, promotion, and course assignment decisions. While research generally agrees that teaching evaluations offer an effective and meaningful way to inform these decisions, often such data are misused, misinterpreted, or overused. Some institutions use student ratings data as the sole criterion for evaluating teaching effectiveness, and these institutions often use only global items on student ratings forms to construct their evaluation. Such misuse can breed distrust between faculty and administrators, resentment on the part of instructors for evaluations, and hinder other formative uses of these data.

Compare this to:

Teaching evaluations are commonly considered for summative purposes, including tenure, merit increase, retention for non-tenured faculty, promotion, and course assignment decisions.  While research generally agrees that teaching evaluations can be used in an effective and meaningful way to inform these decisions, often such data are misused, misinterpreted, or overused.  Some institutions use student ratings data as the sole criterion for evaluating teaching effectiveness, and moreover, these institutions often use only global items on student ratings forms to construct their evaluation.  Such misuse can breed distrust between faculty and administrators, resentment on the part of instructors for evaluations, and hinder other formative uses of these data.

From Agbetsiafa & Aghimien article, page 45:

Instead, researchers recommend that the focus of student evaluation of teaching should be on desired educational outcomes and whether these outcomes are being achieved or not. In this regard, if student ratings forms are to be used, the instruments must be subjected to rigorous validity tests and analysis. In addition, the student-rating data is best used in combination with other criteria in order to provide a better assessment of teaching, which is a multidimensional construct. (See: Bell et al, 1994 Boex, 2000, White, 1995, Hobson & Talbot, 2001; Hoyt & Pallett, 1999; Theall & Franklin, 2001; Kulik, 2001; McKeachie, 1997; Bain, 2004).

Compare this to:

Instead, researchers recommend that when such data are going to be used for summative purposes, various stakeholders, including administrators, faculty and students, should collaborate in determining a proper evaluation system.  The focus of evaluation should be on desired educational outcomes and whether these outcomes are being met.  If student ratings forms are to be used, the instruments must be subjected to rigorous validity tests and analysis.  Further, student rating data should be used in combination with other criteria in order to provide a better assessment of teaching, which is inherently a multidimensional construct.  Multiple sources of data collected over a span of time and covering a variety of courses taught would be most effective in informing summative decision-making. (Hobson & Talbot, 2001; Hoyt & Pallett, 1999; Theall & Franklin, 2001; Kulik, 2001; McKeachie, 1997; 2002; Bain, 2004)

From Agbetsiafa & Aghimien article, page 45:

Using evaluations to inform instructors of their teaching effectiveness and to aid them in improving or enhancing their teaching constitute the formative purposes of teaching evaluations. When used to inform teaching practices, specific dimensions of teaching must be identified and focused upon in order to bring about change. Research indicates that evaluations are most effective in improving teaching when faculty members understand and value the importance of such processes, and an institutional and departmental culture that supports and respects teaching is evident.

Compare this to:

Using evaluations to inform instructors of their teaching effectiveness and to aid them in improving or enhancing their teaching constitute the formative purposes of teaching evaluations.  When used to inform teaching practices, specific dimensions of teaching must be identified and focused upon in order to bring about change.  Research indicates that evaluations are most effective in improving teaching when faculty members understand and value the importance of such processes, and an institutional and departmental culture that supports and respects teaching is evident. 

From Agbetsiafa & Aghimien article, page 45:

Several colleges and universities have begun using portfolios, peer observation, self-review, and more qualitative approaches to improve teaching. Similarly, recent establishment of faculty development centers on many campuses reveals a trend toward investing in the formative uses of evaluations. See, White, 1995, Hobson & Talbot, 2001; Hoyt & Pallett, 1999; Theall & Franklin, 2001; Kulik, 2001; Johnson & Ryan, 2000; Hativa, 1995; Bain, 2004.

Compare this to:

Studies have emerged showing how institutions and individual faculty members have begun using evaluations, consultations, and portfolios to qualitatively improve instruction.  When faculty are well informed about the purposes of evaluation, much of their anxiety dissipates and willingness to learn from student feedback increases.  (Sojka, Gupta, & Deeter-Schmetz, 2002; Hativa, 1995; Gallagher, 2000; Bain, 2004)

From Agbetsiafa & Aghimien article, page 45:

Some of the myths about the usefulness of student ratings begin from faulty research studies, conflicting findings within the research literature, or reluctance on the part of some administrators and faculty to evaluate and be evaluated, respectively. Some common myths include students are not able to make informed and consistent judgments about their instructors; student ratings are essentially a popularity contest; students cannot make accurate judgments unless they have been away from the course for a while; student ratings are negatively related to student learning; student ratings are based upon expected grade in course.

Compare this to:

Many myths exist about the usefulness of student ratings.  Some of these myths originate from faulty research studies, conflicting findings within the research literature, or reluctance on the part of administrators and faculty to evaluate and be evaluated, respectively.  Some common myths of student evaluations of teaching include: students are not able to make informed and consistent judgments about their instructors; student ratings are essentially a popularity contest; students cannot make accurate judgments unless they have been away from the course for a while; student ratings are negatively related to student learning; student ratings are based upon expected grade in course.

From Agbetsiafa & Aghimien article, page 45:

While these myths have been adequately disproved by research, some criticisms of student-ratings of teaching have been long-standing and not resolved. These criticisms center on issues of validity and reliability, and factors that may bias teaching evaluations, including, student, course, and instructor characteristics. For more discussion of these, see Wetzstein and Broder, 1985, Hobson & Talbot, 2001; Aleamoni, 1987; Theall & Franklin, 2001; Kulik, 2001; McKeachie, 1997; Bain, 2004.

Compare this to:

While the above myths have been adequately disproved by research, some criticisms of SET’s have been long-standing and not resolved.  These criticisms center on issues of validity and reliability, and factors that may bias teaching evaluations, including, student, course, and instructor characteristics. (Hobson & Talbot, 2001; Aleamoni, 1999; Theall & Franklin, 2001; Kulik, 2001; McKeachie, 2006; Bain, 2004).

From Agbetsiafa & Aghimien article, page 45:

Reliability refers to the consistency of ratings among different raters and the stability of such ratings over time. Studies by Hobson & Talbot, 2001; Aleamoni, 1999; Marsh & Roche, 1997 conclude that student ratings of teaching show an acceptable level of consistency, or inter-rater reliability, given a class size of at least 15. The level of consistency among raters increases as class size increases.

Compare this to:

Reliability refers to the consistency of ratings among different raters and also the stability of such ratings over time.  Research has shown that student ratings show an acceptable level of consistency, or inter-rater reliability, given a class size of at least 15.  The level of consistency among raters increases as class size increases.  Longitudinal studies and studies of alumni ratings of an instructor/course have found that ratings show high levels of stability over time.  Further, cross-sectional studies show that student ratings reliably reflect instructor versus course effectiveness. (Hobson & Talbot, 2001; Aleamoni, 1999; Marsh & Roche, 1997)

From Agbetsiafa & Aghimien article, pages 45-46:

However, other researchers like Aleamoni, 1999; Theall & Franklin, 2001; Marsh & Roche, 1997; D'Apollonia & Abrami, 1997; and McKeachie, 1997 and critics of student-ratings have suggested numerous factors which may bias student ratings of teacher effectiveness including: class size, grade leniency, instructor personality, gender, course workload, time that class meets, and type of class, including the academic discipline and required/elective status of class. For each of these factors, research has been somewhat inconclusive, with some studies asserting a positive, negative, or no relationship between the variables.
Understanding the potential relationships, however, colleges, universities, and researchers have begun controlling for certain student and course characteristics before examining student ratings.

Compare this to:

Researchers and critics of SET’s have suggested numerous factors which may bias student ratings of teacher effectiveness including: class size, grade leniency, instructor personality, gender, course workload, time that class meets, and type of class, including the academic discipline and required/elective status of class.  For each of these factors, research has been somewhat inconclusive, with some studies asserting a positive, negative, or null relationship between variables. Understanding the potential relationships, however, institutions and researchers have begun controlling for certain student and course characteristics before examining student ratings. (Aleamoni, 1999; Theall & Franklin, 2001; Marsh & Roche, 1997; d’Apollonia & Abrami, 1997; McKeachie, 1997)

From Agbetsiafa & Aghimien article, page 46:

Student rating forms generally contain both global and overall rating items and specific items, which assess specific aspects of the instructor and course. Research results on the value of both of these types of items are mixed.
Some correctly argue that teaching is multi-dimensional and therefore requires specific items in accurately evaluating different aspects of instruction. Others show that when specific items are factor analyzed, they essentially reduce to one or two items that are global in nature. Studies also reveal that responses on specific and global items are highly correlated. With regard to the uses of these types of items, researchers warn against making summative decisions based solely on ratings on global items. In addition, formative purposes seem better informed by having data on specific areas that faculty can target in order to improve their teaching. For more discussion see, Gallager,2000; D'Apollonia & Abrami, 1997; Young & Shaw, 1999; and Bain, 2004.

Compare this to:

Student rating forms generally contain both global (or overall rating) items and specific items, which assess specific aspects of the instructor and course.  Research is split on the value of both of these types of items.  Some argue that teaching is multi-dimensional and therefore requires specific items to accurately assess different facets of teaching.  Others show that when specific items are factor analyzed, they essentially reduce down to one or two items that are global in nature.  Studies also reveal that responses on specific and global items are highly correlated.  

With regard to the uses of these types of items, researchers warn against making summative decisions based solely on ratings on global items.  In addition, formative purposes seem better informed by having data on specific areas that faculty can target in order to improve their teaching. (Gallager, 2000; d’Apollonia & Abrami, 1997; Young & Shaw, 1999; Bain, 2004). http://www.oid.ucla.edu/publications/evalofinstruction/eval6/

From Agbetsiafa & Aghimien article, page 46:

Broadly, factor analysis enables the definition of an underlying or latent structure in a data matrix or data set. It facilitates the analysis of the structure of the interrelationships (correlations) among a large number of variables by defining a set of common underlying dimensions, usually called factors. Thus, it is possible to reorient the data so that the first few dimensions account for as much of the available information as possible. If there is much (or any) redundancy in the data set, then it is possible to account for most of the information in the original data with a considerably reduced number of dimensions.

Compare this to:

Broadly, factor analysis, or more particularly in this case, principal components analysis, enables the definition of an underlying or latent structure in a data matrix or data set. It facilitates the analysis of the structure of the interrelationships (correlations) among a large number of variables by defining a set of common underlying dimensions, usually called factors. Thus, it is possible to reorientate the data so that the first few dimensions account for as much of the available information as possible. If there is much (or any) redundancy in the data set, then it is possible to account for the most of the information in the original data with a considerably reduced number of dimensions.

From Agbetsiafa & Aghimien article, page 50:

Under methods, a number of active learning practices as opposed to passive learning should be listed in the instrument. The second dimension-inventories of good practice in undergraduate education (Chickering and Gamson, 1987) which emphasizes active learning as well as student time on task and other practices. Time on task means that students need to spend the time and make the commitment necessary to prepare for class and assignments.

Compare this to:

Similarly, an educators’ conference at the Wingspread Conference Center in Racine, Wisconsin, in 1986 resulted in the “Inventories of Good Practice in Undergraduate Education,” which emphasized the need for active learning, as well as student time on task and other practices (Chickering & Gamson, 1987). Time on task means that students need to spend the time and make the commitment necessary to prepare for class and assignments.

From Agbetsiafa & Aghimien article, page 50:

The third dimension is course outcomes. Numerous higher education reports discussed the need to focus on student learning. Furthermore, many accrediting organizations have called on colleges and universities to measure student learning and other outcomes of instruction in their self-studies. Among the items to include are ratings of student learning, their independent thinking skills, and other broad outcomes that students attribute to the particular course they are evaluating.

Compare this to:

The third area added, Course Outcomes, reflects still another emphasis in evaluation during the past decade or so. Numerous higher education reports, including the two mentioned above, discussed the need to focus on student learning. The various accrediting agencies have also called on institutions to measure student learning and other outcomes of instruction in their self studies. Items added to the SIR II assessed students’ ratings of their learning, their independent thinking, and other broad outcomes that students attributed to the particular course they were evaluating.

No comments:

Post a Comment