If you have a known standard for each rating, you can assess the correctness of all appraisers' ratings compared to the standard. With ordinal data of 3 or more levels, you can calculate Kendall's correlation coefficient to compare the multiple ratings with the standard.

For the fabric data, Kendall's coefficient is the average of Kendall's rank-order correlation coefficients across trials conducted by all appraisers. Kendall's rank-order correlation coefficient for each trial expresses the degree of association between ratings from the trial and the standard.

Kendall's correlation coefficient uses information about relative ratings and is sensitive to the seriousness of the misclassification. For example, pie crust crispiness is rated on a 1-5 scale. The consequences of misclassifying a perfectly crisp pie crust (rating = 5) as soggy (1) are more serious than misclassifying it as mostly crisp (4).

Kendall's coefficient can range from -1 to 1. Positive value indicates positive association and negative value indicates negative association. In addition, the higher the magnitude, the stronger the association.

Use Kendall's correlation coefficient and their p-values to choose between two opposing hypotheses, based on your sample data:

·    H0: There is no association between ratings of all appraisers and the known standard

·    H1: Ratings by all appraisers are associated with the known standard

The p-value provides the likelihood of obtaining your sample, with its particular Kendall's correlation coefficient, if the null hypothesis (H0) is true. If the p-value is less than or equal to a predetermined level of significance (a-level), then you reject the null hypothesis and claim support for the alternative hypothesis.

For the fabric data, with a = 0.05, p = 0.0000. The ratings for all appraisers are significantly associated with one another and the known standard.