If each appraiser provides two or more ratings for the same unit, you can assess the consistency of each appraiser's ratings across the trials. With ordinal data of 3 or more levels, you can calculate Kendall's coefficient of concordance.

Kendall's coefficient of concordance expresses the degree of association among the multiple ratings made by an appraiser. It uses information about relative ratings and is sensitive to the seriousness of the misclassification. For example, pie crust crispiness is rated on a 1-5 scale. The consequences of misclassifying a perfectly crisp pie crust (rating = 5) as soggy (1) are more serious than misclassifying it as mostly crisp (4).

Kendall's coefficient of concordance can range from 0 to 1. The higher the value of Kendall's, the stronger the association.

Use the p-values to choose between two opposing hypotheses, based on your sample data:

·    H0: There is no association among multiple ratings made by an appraiser

·    H1:The ratings are associated with one another

The p-value provides the likelihood of obtaining your sample, with its particular Kendall's coefficient of concordance, if the null hypothesis (H0) is true. f the p-value is less than or equal to a predetermined level of significance (a-level), then you reject the null hypothesis and claim support for the alternative hypothesis.

For the fabric data, with a = 0.05, for all appraisers, p < 0.05 (0.0011, 0.0012, 0.0013, 0.0012), so you can reject the null hypothesis. The print quality ratings are significantly associated with one another.