The Person-Centered e-Learning Pattern Repository

Peer-Evaluation

Package: Evaluation

Intent?

Peer-evaluation fosters active participation and engagement of participants, as they take on the roles of peers as evaluators in Evaluation scenarios.

Motivation?

Typically, in conventional course settings, students may not be interested in or come to know work contributed by other students and teams, as long as they get their own work right. Often they do not even know what other students and teams are working on. Only students who come equipped with prior personal interest in the course's subject are likely to be motivated to engage with their own and other students' contributions. Additionally, it is often hard to accomplish for instructors to assess fair grades to the students, taking into account the diverse topics and interests.

In peer-evaluation scenarios participants evaluate certain contributions of each other. The infrastructure for students to communicate the evaluation to the instructor is provided, e.g., via a web form. There are several variants of implementing this pattern in a course. They can be derived from the questions that specify the peer-evaluation process as shown in Table 1. The questions in the left-hand side column aim at certain independent variables of a peer-evaluation solution. The right-hand side column shows possible answers to the questions, which may be adapted by the pattern user and combined arbitrarily to assemble one feasible solution.

Table 1: Possible solutions of implementing Peer-Evaluation.

VariablePossible solutions
Who are the evaluators?Single participants or teams
What is the evaluation method?Written (offline/online), Oral
What kind of evaluation is expected?Grades, bonus points, structured or unstructured comments, or any combination of these
Are the evaluations visible to peers?No, anonymously, or with name

What is potentially subject to evaluation depends largely on the type of learning activity that produces the evaluation targets, e.g. designing a website will impose different criteria on evaluation than writing an essay. Additionally, the instructor may define certain aspects and criteria to be evaluated, e.g. focusing on content, ignoring writing style. Imposing criteria even seems recommendable, as some studies (Cho & Schunn, 2003; Mockford, 1994) stress the rather moderate validity of peer-evaluation results, which means that participants often evaluate their peers using different criteria than the instructor.

As can bee seen in Table 1, there exist a host of possible peer-evaluation solutions, each feasible, but with different consequences and meaning. For example, consider the following two peer-evaluation solutions to a course with 20 students, organized in teams of four. Say, there were three assignments each team had to work out.

  • Let each team write evaluations of at least two other dedicated teams: This scenario will result in (five teams) x (two dedicated teams) = 10 evaluations which the instructor has to consider in the evaluation process.
  • Let each participant grade and briefly comment the main contribution of every other student: Here the instructor will have to take into account (20 students) x (19 peer students) x (one final assignment) = 381 peer-evaluations, which is organizationally nearly impracticable, unless the instructor receives the evaluations in some structured form.

The revision of a participant's evaluation by the instructor may be postponed until the Instructor-Evaluation of the respective participant, which may reduce the overhead of considering large numbers of peer-evaluations. Both of the above scenarios show extremes, but they reveal the impact of peer-evaluation design decisions on organization and the instructor's schedule in the Assessment Phases.

On the students' side, peer-evaluation has many positive effects:

  • Motivation is likely to rise because the students know their work will be reviewed and evaluated by peers. This situation can produce an atmosphere of positive reciprocal stimulation and competition.
  • Students are given the opportunity to compare their contributions and evaluations to those of others. This can lead to exchange of ideas on the subjects, what itself may arouse deeper interest in the whole subject area.
  • Contributions are seen from different points of view.
  • From the perspective of learning, peer-evaluation using comments helps students improve their performances.

On the instructor's side:

  • Peer-evaluation provides the instructor with insights and perspectives in what the students think about and expect from each other.
  • It supports fair assessment of a grade as it gives broader, more diverse perspectives.
  • Less enjoyable is the organizational overhead peer-evaluation is able to introduce when not properly dealt with. Especially in cases with many evaluations among the students, the instructor may require a tutor or assistant to electronically solicit, collect and prepare grades and comments. The first class solution to this problem is structured delivery of grades and comments, e.g. through a suitable web form that produces and stores data the instructor can process as needed. Given lack of technical platform support or only few participating evaluators, simpler means of collecting and processing evaluations may be employed, such as e-mail.
  • Implementations of this pattern have shown that soliciting unstructured comments (maybe in connection with a grade or bonus points) yield more perspectives and more meaningful opinions on the evaluation targets than a structured evaluation template approach (see the Examples section).

Scenario?

ItemDescription
Define evaluation targetsThe instructor has to decide which contributions of the participants he or she wants to be evaluated as evaluation targets. For example, in a Learning Contracts scenario, the elaborations in the scope of a participant's contract may be defined as evaluation targets. There is a web template for online evaluation.
Assign evaluators to targetsThe instructor may optionally assign peers (evaluators) to evaluation targets for evaluation. Usually, it is more useful to let the participants choose their evaluation targets, as the meanings und insights of their evaluations certainly increase when they are free to evaluate targets based on their own personal interests. Alternatively, the instructor may assign certain evaluation targets to each evaluator.
Publish peer-evaluation detailsAll details the instructor has defined so far have to be made clear to the participants along with guidelines and criteria for evaluations. Apart from the whole evaluation procedure, it is particularly important to define what kind of evaluations and comments the instructor expects (see Table 1).
Choose targetIf the instructor did not assign certain targets to be evaluated by the peers, the participants may now freely choose evaluation targets for evaluation. The instructor should define the expected number of targets each participant has to evaluate.
Evaluate peers based on targetsThis activity is inherited from Evaluation. It is supported by a web template.
Publish evaluationsDepending on what has been negotiated with the participants, the instructor may Publish the evaluations to the participants, either anonymously or with name.

Parameters?

Examples?

In any of the examples below, the evaluations were anonymous for the recipients, but transparent to the instructors.

Web Engineering (2003): Peer-evaluation was used in the Learning Contracts scenario in Web Engineering, in the so called Web Engineering Learning License (WELL) project. Each WELL participant had to peer-evaluate at least three contracts elaborated by peer teams. They had to provide a written, unstructured evaluation, and could optionally supply 0 - 5 bonus with each evaluation. Evaluation guidelines were published prior to the start of the evaluation phase. The evaluations were used by the instructors to prepare themselves for the final oral Examination with each participating WELL team. Additionally, the ranking of contracts calculated from bonus points served as input for a subsequently handing over Achievement Awards to contributors of outstanding (i.e., best-evaluated) contracts. The evaluation form can be viewed at: http://elearn.pri.univie.ac.at/patterns/examples/peer-evaluation/we.html

Person-Centered Communication: Each student had to evaluate seminar reports of at least two teams. Thereby the students had to supply a written evaluation, whereas the following criteria were defined as evaluation guidelines:

  • Is the report interesting? Is it readable and well structured?
  • What is appealing, what is not so interesting?
  • Did the authors include own comments and experiences? Do I find these comments interesting?
  • Are there any thought-provoking impulses?

Additionally, the evaluators had to supply 1 - 10 points, whereby 1 = unsatisfying and 10 = very satisfying. The evaluation form can be viewed at: http://elearn.pri.univie.ac.at/patterns/examples/peer-evaluation/pcc.html

Project Management/Soft Skills: Students were organized in five small teams. Each team had to evaluate the work (the sum of all contributions) of all other teams. Each team had elaborated a number of unique, different contributions throughout the course. The following aspects were formulated and published as evaluation criteria:

  • Provided resources: Readability, structure, degree of interests/motivation, applicability as soft skill,
  • Design concept (thread to follow)
  • Implementation of the concept
  • Overall impression that evolved through the team, the subject, and its moderation in the course module.

The evaluation form can be viewed at: http://elearn.pri.univie.ac.at/patterns/examples/peer-evaluation/pmss.html

Evaluation?

As given in Figure 1, empirical investigation in the Web Engineering case study substantiated that peer-evaluation is perceived as very or rather useful by more than 45.6% of the students, while it is considered rather useless or useless by only 21.4% of the participants. About one third of the participants found the procedure just acceptable. However, more supportive is the fact that more than half of the participants valued reading the evaluations of their own contributions as meaningful or very meaningful (52.9% combined as given in Figure 2), while only 19.1% consider this as meaningless.

Figure 1: Students' perception of the usefulness of peer-evaluation in Web Engineering.

Figure 2: Students' valuing of reading evaluations of their own contributions.

Regarding the anonymity of peer-evaluations, the Web Engineering case study revealed that participants rather want to their evaluations to appear anonymous to their peers (47.1% as opposed to 14% opting for non-anonymity; see Figure 3).

Figure 3: Students' opinion on the anonymity of peer-evaluations.

Other Studies

Wen and Tsai (Wen & Tsai, 2003) have investigated students' perceptions of and attitudes toward online peer-evaluation using 4 subscales with responses ranging from 1 = strongly disagree to 5 = strongly agree (note that this reproduction uses the term evaluation instead of assessment and mark/grade):

  • Positive Attitude Subscale (PAS): Peer-evaluation
    • is helpful to my learning
    • makes me understand more about teacher's requirement
    • activities can improve my skills in verbal interaction
    • activities motivate me to learn
    • activities increase the interaction between my teacher and me
    • helps me develop a sense of participation
    • activities increase the interaction between my classmates and me
    • is fair to assess students' performance
  • Online Attitude Subscale (OAS): Online peer-evaluation activities
    • can be timesaving
    • can increase the interaction among classmates
    • can be economical
    • can increase the interaction between the teacher and the students
    • are fair when assessing students' performance
  • Understanding-and-Action Subscale (UAS):
    • Peer-evaluation activities help me to understand what other classmates think
    • The teacher should develop criteria of peer-evaluation activities for students
    • Students should participate in the development of criteria for peer-evaluation activities
  • Negative Attitude Subscale (NAS):
    • I think students should not be responsible for making evaluations
    • Peer-evaluation is time-consuming
    • My evaluation giving to classmates are affected by the evaluations given to me
    • If I receive evaluations worse than expected, then I will give worse evaluations to classmates

The results of this study showed that this questionnaire is a highly reliable instrument to assess students' attitudes toward peer-evaluation regardless of level of education. University students generally perceive peer-evaluation in a positive way. Interesting is the fact that students who have prior experience with peer-evaluation tend to have more positive opinions on this technique when compared to students who have never acted as evaluators. Thus, it seems recommendable to establish this evaluation technique as an integral part of blended learning courses.

A study on online peer-evaluation by Akahori and Kim (Akahori & Kim, 2003), which was conducted in a problem-based learning (PBL) course, revealed some interesting results:

  • Novices (i.e., students) tend to show lower appreciation to problem-solving methods and problem proposals than experts (i.e., instructors). Instead they focus more on comprehensive literature references in a report as well as on the validity of the solution.
  • Peer-evaluation activities highly promote students' motivations to learn by themselves as well as by modeling from their peers.
  • Students show high level of interest and concern towards peer-evaluation, which is especially due to the unusual experience of coming to review peers' reports.
  • Contributions that are subject to peer-evaluation are of higher quality (higher score) than reports evaluated traditionally.
  • Feedback included many comments about learning how to improve report writing and about how to distinguish different qualities of reports.

Two studies in project-based learning (PBL) by Sluijsmans and colleagues (Sluijsmans et al., 2001) show rather diverse results. Subject to peer-evaluation was the PBL process itself, not its products:

References?

Akahori, K., & Kim, S. M. (2003). Peer Evaluation Using the Web and Comparison of Meta-cognition between Experts and Novices. Proceedings of ED-MEDIA 2003 - World Conference on Educational Multimedia, Hypermedia & Telecommunications, Honolulu, HI, USA.

Cho, K., & Schunn, C. D. (2003). Validity and Reliability of Peer Assessments with a Missing Data Estimation Technique. Proceedings of ED-MEDIA 2003 - World Conference on Educational Multimedia, Hypermedia & Telecommunications, Honolulu, HI, USA.

Mockford, C. D. (1994). The use of peer group review in the assessment of project work in higher education. Mentoring and Tutoring, 2 (2), 45-52.

Sluijsmans, D. M. A., Moerkerke, G., van Merrienboer, J. J. G., & Dochy, F. J. R. C. (2001). Peer Assessment in Problem Based Learning. Studies in Educational Evaluation, 27, 153-173.

Wen, M. L., & Tsai, C.-C. (2003). College Students' Perceptions of and Attitudes Toward Online Peer Assessment. Proceedings of ED-MEDIA 2003 - World Conference on Educational Multimedia, Hypermedia & Telecommunications, Honolulu, HI, USA.

User comments
Reply: XwIBIgyWXzYM
by Eflenmar, on Friday, September 20, 2013, 8:08 AM
BUY MITCHELL STARC we have enough money to bid on him (since most of the big fraheniscs have little budget left, this is our chance) I would suggest getting him and a couple of good local players like Jiwanjot Singh and the Kaul brothers. That plus the talent in the likes of Mandeep Singh and Gurkeerat Singh will really strengthen the team. Shaun Marsh is also in good form, so out foreign players can be Marsh, Hussey, Mahmood and Starc. http://pvrwzoq.com [url=http://gwcxttcv.com]gwcxttcv[/url] [link=http://gvffdtc.com]gvffdtc[/link]
Reply: dLxpIwkFWtJstLYiDEZ
by Janaye, on Wednesday, September 18, 2013, 2:14 PM
Articles like these put the consumer in the driver seat-very imatotpnr. http://uspwfssb.com [url=http://pbfqnnlurt.com]pbfqnnlurt[/url] [link=http://ntjodn.com]ntjodn[/link]
Reply: YsBhwYUKaBSvfI
by Vicki, on Wednesday, September 18, 2013, 11:57 AM
Very true! Makes a change to see sooemne spell it out like that. :) http://hclsqtwyt.com [url=http://gpwgaimujmw.com]gpwgaimujmw[/url] [link=http://ujvwoh.com]ujvwoh[/link]
Reply: CvhGRLqcFVjInmdXqCM
by Jase, on Tuesday, September 17, 2013, 11:09 PM
Thanks for that! It's just the answer I <a href="http://zidmde.com">needde.</a>
Reply: dYzScnAoViLLhLZeVwp
by Amirkhg, on Monday, September 16, 2013, 8:06 PM
Begun, the great internet <a href="http://nnwwpirw.com">edauction</a> has.
Reply: AcrrQFlsOR
by Meso, on Monday, September 16, 2013, 10:40 AM
Reading posts like this make surfing such a plearuse http://kwlerufpxq.com [url=http://aibspnw.com]aibspnw[/url] [link=http://lvgehlpcl.com]lvgehlpcl[/link]
Reply: atFjzsiuyCfmZ
by Ajit, on Monday, September 16, 2013, 2:44 AM
I think that's what was meant (though I can't speak for the author). There are some<a href="http://lplqevehxi.com"> motnems</a> of misogyny is the game itself, but it's presented in a world where April is generally supported. In any world there will be one or two skeazeballs, I suppose. Something I wanted to broach in another post.As for availability? You might find it in a PC gaming establishment, though I'm not sure if that would be entirely productive. If you don't mind digital distribution, I know it's available on GoG (though given their recent issue with the t-shirt, it is something that might need some thought depending) and Steam.
Reply: khilnMqUtfnLJE
by Bertha, on Monday, September 16, 2013, 12:10 AM
One more issue is really that video ganmig became one of the all-time main forms of excitement for people of nearly every age. Kids have fun with video games, plus adults do, too. The particular XBox 360 is among the favorite ganmig systems for people who love to have a lot of activities available to them, as well as who like to play live with other people all over the world. Many thanks for sharing your notions. http://jozlkcj.com [url=http://moutkwik.com]moutkwik[/url] [link=http://rchjdwpfa.com]rchjdwpfa[/link]
Reply: suPMdiXPogxyDDeIz
by Felipe, on Sunday, September 15, 2013, 8:26 PM
I am commenting to let you know what a fine<a href="http://fgetcvnpv.com"> dioesvcry</a> my wife's child undergone visiting your web site. She realized some issues, which include what it is like to possess an amazing teaching spirit to have many people really easily know just exactly various extremely tough things. You undoubtedly surpassed our own desires. Many thanks for churning out such priceless, dependable, informative as well as cool tips on your topic to Gloria.
Reply: EGlsVhPuEOTjTRaZzA
by Christine, on Sunday, September 15, 2013, 10:13 AM
Hey Karen & Violet I guess you only count if you actually try to cnmoemt on every post like I do. That way she knows you're reading!I get the feeling I got to partake in the newly updated polls since I did not get redirected to another site.I ended up choosing other because I use 2 of the three choices in the second poll.
Reply: JSuNWjEtYgioM
by Knut, on Sunday, September 15, 2013, 10:09 AM
I in addition to my beduids ended up reading the good tips located on your web blog then all of a sudden came up with a horrible suspicion I never thanked the web blog owner for them. The ladies were definitely for this reason happy to read them and now have absolutely been tapping into these things. Many thanks for truly being considerably thoughtful and also for choosing this form of brilliant issues most people are really wanting to learn about. Our own honest regret for not expressing gratitude to you sooner. http://atsjcjhzpfv.com [url=http://jqaleop.com]jqaleop[/url] [link=http://jtvfrcnlruy.com]jtvfrcnlruy[/link]
Reply: VjRpmzZnwhHn
by Gengo, on Sunday, September 15, 2013, 6:33 AM
I couldn't agree more about <a href="http://rlulyiin.com">geticrrias</a>. I miss sitting with some of the elderly individuals I used to help and they would tell me stories for hours about the things that they witness when they were younger, and the lives that they lived up to the point that they needed care. I have always felt that the elderly know how to live their lives better than anyone, and they for sure never take it for granted! Sometimes the people taking care of them are not good people though, and it always breaks my heart to see someone talking badly to an elderly man or woman.I like how you made a switch from little tiny babies that have no real experience with life, to elderly men and women who have lived life to its fullest and still have more life to share with others. Such opposites!
Reply: aksBCRjkOzpc
by Open, on Saturday, September 14, 2013, 12:21 AM
For me, choosing a mdiceal specialty is easy. I was lucky enough to have the opportunity to work as a Dermatologist Tech prior to moving to Alaska. What an amazing field! No two cases are the same and more than one type of procedure can be performed in a day. Working for a dermatologist provides the benefits of both a private office setting and hours along with the experience of surgeries and unique treatments. Cyst removals and cancer treatments became the most interesting procedures to me and I can not wait to get back into the field!There are not many positions or specialties I would turn down when it comes to the mdiceal field. If I had to pick one, however, it would have to be gynecology. Why would this be the field I turn down? I am honestly not sure. I must be lacking the gene the would cause me to be interested in the subject! For whatever reason I would choose to work anywhere else before choosing gynecology. Specialty clinics are like sushi I guess. How do you know you don't like it if you've never tried it?!
Reply: jjTgszjYjGrWWoROrjkE
by Cayetano, on Friday, September 13, 2013, 9:27 PM
I precisely<a href="http://ljbqolb.com"> watend</a> to thank you so much yet again. I am not sure the things I could possibly have made to happen in the absence of these solutions provided by you regarding such field. It has been an absolute daunting setting in my opinion, however , understanding this well-written technique you treated the issue forced me to leap over contentment. Now i am grateful for your help and in addition sincerely hope you comprehend what an amazing job you're putting in instructing most people through your webblog. Probably you haven't got to know any of us.
Reply: UCjuZgRZDNRn
by Kayoko, on Friday, September 13, 2013, 3:17 AM
is apt to the few of occasions, prpiesiositdon arm in his hands look salt curtailed, caught in the armpit since it seems ritzy; carrying limerick corner, arms drooping as expected, is inwards handsome.Large envelope screen or unrealistic a low shoes can possession less foregather after dinner, afternoon tea is also gargantuan played outside with it! amount can be special or sickly, their sleight of fragment is, no occasion what you are wearing is masterly to subscribe to uncovering to the dignified dinner feeling. Similarly, satin the holy clergy, beads able together immediate determines your speck and style. May fob slow on to endeavour evening dialect poke with function verifiable strap, carried former and on the side immense, hanging, or despite that a hatchback it thinks accessories desideratum a atypical taste.
TgFSjgxwGWUyrLDTGyk
by hqeuygxpgj, on Tuesday, June 11, 2013, 2:00 PM
MnrwZk <a href="http://lwyartgkgobc.com/">lwyartgkgobc</a>
rpUevbmVZi
by fttjvmakb, on Sunday, June 09, 2013, 6:18 AM
KpXZ2d , [url=http://uzsvowamctso.com/]uzsvowamctso[/url], [link=http://npnycibokxwp.com/]npnycibokxwp[/link], http://godhzvpfinbx.com/
MIECuSZrZHjsSBh
by Lau, on Wednesday, June 05, 2013, 3:53 PM
i dont really uetarsdnnd why mannargudi is full of creative guys.i wonder? my new friend sunder alais sundar raja cholan is also a mannargudian(i named it him). suprisngly he is very great poet. i mean it. recently he has written a aanthai to lord shiva. i suppose there is something spl in mannargudi.