Online Peer Review System

From Mosuma
Jump to: navigation, search

Accessing the online peer review system

  • Log into the system as soon as possible to fix any problems with your accounts
    • username is the part before the 'at' sign, e.g. the part within curly braces
      • {newstyleusername}@e.ntu.edu.sg
      • {oldstyleusername}@ntu.edu.sg
      • username is NOT {selfcreatedalias}@pmail.ntu.edu.sg
    • email askychang@ntu.edu.sg if the system does not recognize your username or you have problems
    • let me know where (in/out Singapore, in/out NTU) you are accessing the system
  • username={yourntuusername in all *lowercase*, e.g., praveena1}
  • password={please *click "forgot password"* and enter your ntu email, e.g., praveena1@e.ntu.edu.sg}
  • You will receive an activation email at your NTU email to activate your account, remember to add ":9999" in the activation link if you are accessing within NTU hostel/office/lab), e.g.,
    • if you *originally can only access after adding 9999*, when you receive this in your activation email:
http://peer.mosuma.net/reset/3r-37b-6c60d5d98381185bdddafdafdsa/.

you need to change to this (*:9999* is added after *"mosuma.net"*):

http://peer.mosuma.net:9999/reset/3r-37b-6c60d5d98381185bdddafdafdsa/. 

Please *do not* email me the peer review form, unless you have excruciating difficulties in accessing the online peer review system.

When voting

  • Your review is confidential
  • Comments required to justify any **non-zero** scores, otherwise submissions will be ignored
    • Discussion of votes will likely lead to a biased voting behaviour, you could discuss general qualities of a person, but not the vote intention of fellow team members
    • Think very carefully before you assign a "-1" score to any person
    • However, team members who performs *far below* your expectation should get a "-1"
    • If you try to be compassionate by not giving someone who deserves a '-1' a '-1', do not regret when his final grade is same or just marginally worse off than yours, e.g., you assign '0' to a slacker, he gets a 'B' for CID, and you get 'B+', don't come to me afterwards to tell me how he should deserve a lower grade and you were trying to play nice.
    • Every team member assigning +1 to every member in the group will have the same effect of assigning zero to every one. I will be normalizing scores to zero mean and unit variance. Please be selective if you want your votes to have an actual effect.
    • Be fair and impartial
    • Do not let your emotions affect your judgement, e.g., do not assign a "+1" to a slacker who looks gorgeous or is sweet to you.
  • Scores from everyone will be added up
    • Max Score = (teamsize)
    • Min Score = - Max Score
    • -1 will be added to your total score if you do not submit a peer review
    • +1 will be added to your total score if you submit a peer review, i.e., 1 out of (Max Score) will come from your participation in the peer voting.

Sample peer review (Typical)

Person Anna Bob Calvin David Ester
Anna +1* +1 0 -1 0
Bob 0 +1* 0 0 0
Calvin -1 -1 +1* -1 -1
David 0 0 0 -1 0
Ester +1 +1 +1 +1 +1*
∑ Votes 1 2 2 -2 0
Sca_Votes 0.2=1/5 0.4=2/5 0.4=2/5 -0.4=-2/5 0
Std_Votes .27 .94 .94 -1.74 -0.4
CID_Score P+0.27m P+0.94m P+0.94m P-1.74m P
  • Everyone who logged in to submit his/her peer evaluation will receive a +1 for his/her own score
  • Sca_Votes = "Scaled votes" to [-1,1] by dividing total votes by 5 (which is the maximum votes in this example)
  • Std_Votes = votes standardized to zero mean and unit variance within team
  • CID_Score = Project_Score (P) + Std_Votes * m
    • Project score (P) is the score assigned to the team's project by the examiner
    • m is a secret adjustable parameter (ranging from 5-10)
    • if your Sca_Votes >=0, then your final score will not be penalized
    • if Your Sca_Votes <0, then your final score will be penalized according to the above formula

Analysis of Individuals

  • Anna is the team leader, and she carefully rated each team member
  • Bob too lazy to rate his peers, he simply login to the system to earn his +1, and assign zero to everyone (which is the default)
  • Calvin hates everyone in the group, he rated everyone -1
  • David did not login to the online-system, so his default score for everyone is zero, but his own score has a "-1" penalty for not even logging in to the system
  • Ester thinks that everyone is great, she assigns +1 to everyone.

Sample peer review (everyone rated +1)

Person Anna Bob Calvin David Ester
Anna +1 +1 +1 +1 +1
Bob +1 +1 +1 +1 +1
Calvin +1 +1 +1 +1 +1
David +1 +1 +1 +1 +1
Ester +1 +1 +1 +1 +1
∑ Votes 5 5 5 5 5
Sca_Votes 1=5/5 1=5/5 1=5/5 1=5/5 1=5/5
Std_Votes 0 0 0 0 0
CID_Score P P P P P
  • Each member's CID_Score = P (Project_Score)
  • If everyone in your team is as great as the votes, than the greatness would have been reflected in the wonderful project score P, and thus no adjustments are needed

Sample peer review (everyone rated 0)

Person Anna Bob Calvin David Ester
Anna +1 0 0 0 0
Bob 0 +1 0 0 0
Calvin 0 0 +1 0 0
David 0 0 0 +1 0
Ester 0 0 0 0 +1
∑ Votes 1 1 1 1 1
Sca_Votes 0.2=1/5 0.2=1/5 0.2=1/5 0.2=1/5 0.2=1/5
Std_Votes 0 0 0 0 0
CID_Score P P P P P
  • Each member's CID_Score = P (Project_Score)
  • If everyone in your team is as mediocre as the votes, than the mediocrity would have been reflected in the mediocre project score P, and thus no adjustments are needed

Sample peer review (everyone rated -1)

Person Anna Bob Calvin David Ester
Anna +1 -1 -1 -1 -1
Bob -1 +1 -1 -1 -1
Calvin -1 -1 +1 -1 -1
David -1 -1 -1 +1 -1
Ester -1 -1 -1 -1 +1
∑ Votes -4 -4 -4 -4 -4
Sca_Votes -0.8=-4/5 -0.8=-4/5 -0.8=-4/5 -0.8=-4/5 -0.8=-4/5
Std_Votes 0 0 0 0 0
CID_Score P P P P P
  • Each member's CID_Score = P (Project_Score)
  • If everyone in your team is as underperforming as the votes, than the poor contributions would have been reflected in the poor project score P, and thus no adjustments are needed