If you have a known standard for each rating, you can assess the correctness of each appraiser's ratings across trials. With ordinal data of 3 or more levels, you can calculate Kendall's correlation coefficient to examine the association between the ratings and the standard.

For the fabric data, Kendall's coefficient is the average of Kendall's rank-order correlation coefficients across all trials conducted by one appraiser. For each trial, Kendall's rank-order correlation expresses the degree of association between appraisers' ratings and the standard. Kendall's correlation coefficient uses information about relative ratings and is sensitive to the seriousness of the misclassification. For example, pie crust crispiness is rated on a 1-5 scale. The consequences of misclassifying a perfectly crispy pie crust (rating = 5) as soggy (1) are more serious than misclassifying it as mostly crisp (4).

Kendall's coefficient can range from -1 to 1. A positive value indicates positive association and a negative value indicates negative association. In addition, the higher the magnitude is, the stronger the association.

Use the p-values to choose between two opposing hypotheses, based on your sample data:

·    H0: There is no association between the ratings of each appraiser and standard

·    H1: Ratings by each appraiser are associated with the standard

The p-value provides the likelihood of obtaining your sample, with its particular Kendall's correlation coefficient, if the null hypothesis (H0) is true. If the p-value is less than or equal to a predetermined level of significance (a-level), then you reject the null hypothesis and claim support for the alternative hypothesis.

For the fabric data, with a = 0.05, for all appraisers, p = 0.0000, so you can reject the null hypothesis. Rating agreement for print quality are significantly different from those that would be achieved by chance.