Table 2 Inter-rater agreement for annotations of the EAR dataset (Study 2).
 | Sample 2A | Sample 2B | Sample 2C | Sample 2D | All Samples |
---|---|---|---|---|---|
Instances | 17,307 | 9824 | 16,874 | 6953 | 50,961 |
Annotators | 4 | 4 | 4 | 3 | 10 |
Label | Prevalence-adjusted bias-adjusted Kappas | ||||
---|---|---|---|---|---|
Moral | 0.73 | 0.75 | 0.85 | 0.91 | 0.80 |
Care | 0.92 | 0.96 | 0.96 | 0.97 | 0.95 |
Harm | 0.96 | 0.98 | 0.97 | 0.99 | 0.97 |
Fairness | 0.98 | 0.96 | 0.99 | 1.00 | 0.98 |
Cheating | 0.97 | 0.95 | 0.99 | 0.99 | 0.98 |
Loyalty | 0.99 | 1.00 | 0.97 | 0.99 | 0.99 |
Betrayal | 0.99 | 0.99 | 0.99 | 1.00 | 0.99 |
Authority | 0.95 | 0.98 | 0.98 | 0.98 | 0.97 |
Subversion | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
Purity | 0.97 | 0.98 | 0.99 | 0.99 | 0.98 |
Degradation | 0.90 | 0.92 | 0.98 | 0.99 | 0.94 |