Example of Attribute Agreement Analysis
main topic      interpreting results      session command      see also  

An educational testing company is training five new appraisers for the written portion of the twelfth-grade standardized essay test. The appraisers' ability to rate essays consistent with standards needs to be assessed. Each appraiser rated fifteen essays on a five-point scale (-2, -1, 0, 1, 2).

1    Open the worksheet ESSAY.MTW.

2    Choose Stat > Quality Tools > Attribute Agreement Analysis.

3    In Attribute column, enter Rating.

4    In Samples, enter Sample.

5    In Appraisers, enter Appraiser.

6    In Known standard/attribute, enter Attribute.

7    Check Categories of the attribute data are ordered and click OK.

Session window output

Attribute Agreement Analysis for Rating

 

 

Each Appraiser vs Standard

 

 

Assessment Agreement

 

Appraiser   # Inspected  # Matched  Percent       95% CI

Duncan               15          8    53.33  (26.59,  78.73)

Hayes                15         13    86.67  (59.54,  98.34)

Holmes               15         15   100.00  (81.90, 100.00)

Montgomery           15         15   100.00  (81.90, 100.00)

Simpson              15         14    93.33  (68.05,  99.83)

 

# Matched: Appraiser’s assessment across trials agrees with the known standard.

 

 

Fleiss’ Kappa Statistics

 

Appraiser   Response    Kappa  SE Kappa        Z  P(vs > 0)

Duncan      -2        0.58333  0.258199  2.25924     0.0119

            -1        0.16667  0.258199  0.64550     0.2593

            0         0.44099  0.258199  1.70796     0.0438

            1         0.44099  0.258199  1.70796     0.0438

            2         0.42308  0.258199  1.63857     0.0507

            Overall   0.41176  0.130924  3.14508     0.0008

Hayes       -2        0.62963  0.258199  2.43855     0.0074

            -1        0.81366  0.258199  3.15131     0.0008

            0         1.00000  0.258199  3.87298     0.0001

            1         0.76000  0.258199  2.94347     0.0016

            2         0.81366  0.258199  3.15131     0.0008

            Overall   0.82955  0.134164  6.18307     0.0000

Holmes      -2        1.00000  0.258199  3.87298     0.0001

            -1        1.00000  0.258199  3.87298     0.0001

            0         1.00000  0.258199  3.87298     0.0001

            1         1.00000  0.258199  3.87298     0.0001

            2         1.00000  0.258199  3.87298     0.0001

            Overall   1.00000  0.131305  7.61584     0.0000

Montgomery  -2        1.00000  0.258199  3.87298     0.0001

            -1        1.00000  0.258199  3.87298     0.0001

            0         1.00000  0.258199  3.87298     0.0001

            1         1.00000  0.258199  3.87298     0.0001

            2         1.00000  0.258199  3.87298     0.0001

            Overall   1.00000  0.131305  7.61584     0.0000

Simpson     -2        1.00000  0.258199  3.87298     0.0001

            -1        1.00000  0.258199  3.87298     0.0001

            0         0.81366  0.258199  3.15131     0.0008

            1         0.81366  0.258199  3.15131     0.0008

            2         1.00000  0.258199  3.87298     0.0001

            Overall   0.91597  0.130924  6.99619     0.0000

 

 

Kendall’s Correlation Coefficient

 

Appraiser      Coef   SE Coef        Z       P

Duncan      0.87506  0.192450  4.49744  0.0000

Hayes       0.94871  0.192450  4.88016  0.0000

Holmes      1.00000  0.192450  5.14667  0.0000

Montgomery  1.00000  0.192450  5.14667  0.0000

Simpson     0.96629  0.192450  4.97151  0.0000

 

 

Between Appraisers

 

 

Assessment Agreement

 

# Inspected  # Matched  Percent      95% CI

         15          6    40.00  (16.34, 67.71)

 

# Matched: All appraisers’ assessments agree with each other.

 

 

Fleiss’ Kappa Statistics

 

Response     Kappa   SE Kappa        Z  P(vs > 0)

-2        0.680398  0.0816497   8.3331     0.0000

-1        0.602754  0.0816497   7.3822     0.0000

0         0.707602  0.0816497   8.6663     0.0000

1         0.642479  0.0816497   7.8687     0.0000

2         0.736534  0.0816497   9.0207     0.0000

Overall   0.672965  0.0412331  16.3210     0.0000

 

 

Kendall’s Coefficient of Concordance

 

    Coef  Chi - Sq  DF       P

0.966317   67.6422  14  0.0000

 

 

All Appraisers vs Standard

 

 

Assessment Agreement

 

# Inspected  # Matched  Percent      95% CI

         15          6    40.00  (16.34, 67.71)

 

# Matched: All appraisers’ assessments agree with the known standard.

 

 

Fleiss’ Kappa Statistics

 

Response     Kappa  SE Kappa        Z  P(vs > 0)

-2        0.842593  0.115470   7.2971     0.0000

-1        0.796066  0.115470   6.8941     0.0000

0         0.850932  0.115470   7.3693     0.0000

1         0.802932  0.115470   6.9536     0.0000

2         0.847348  0.115470   7.3383     0.0000

Overall   0.831455  0.058911  14.1136     0.0000

 

 

Kendall’s Correlation Coefficient

 

    Coef    SE Coef        Z       P

0.958012  0.0860663  11.1090  0.0000

Graph window output

 

 

Interpreting the results

Minitab displays three assessment agreement tables: Each Appraiser vs Standard, Between Appraisers, and All Appraisers vs Standard. Kappa and Kendall's statistics are included in each of these tables.  Overall, these statistics suggest good agreement. The overall Kendall coefficient between appraisers is 0.966317 (p=0.0000). Kendall's coefficient for all appraisers vs the standard is 0.958012 (p=0.0000).

However, a closer look at the Each Appraiser vs Standard table indicates that Duncan's and Hayes' assessments match poorly with the standard. Holmes and Montgomery, however, matched 15 of 15 assessments, for a percentage match of 100%. Note that each percentage matched is associated with a confidence interval.

The Appraiser vs Standard graph provides a graphical display of the Each Appraiser vs Standard assessment agreement table. You can easily compare the assessment agreements for the five appraisers.

Based on this data, you conclude that Duncan, Hayes, and Simpson need additional training.

Note

The Within Appraiser table is not displayed because each appraiser conducted a single trial.