Original Article

A Modification on Intra Class Correlation Estimation for Ordinal Scale Variable Using Latent Variable Model

Abstract

Background: A common way for computing test-retest reliability is Intra Class Correlation which was developed for continuous variables. But it widely used to assess test-retest reliability in questionnaires with Likert scales. Most of the time consecutive numbers regarded as option labels of a question. If the probability of choosing options be the same, using this method is logic, otherwise it is not. Therefore, in this study a modified estimator of ICC is proposed to improve the estimation of ICC for ordinal scale by using latent variable model.

Method: In this method test-retest answers were considered as bivariate variables and cumulative Probit latent variable model was fitted. A simulation study with N=1500 replicates was conducted to compare the ICC estimations of Likert scale approach with a latent variable approach. Different sample sizes (n=20, 30) was generated with different correlation parameters. The simulations were repeated for questions with 3 and 5 options with different probability of selecting options of a question.  After that the two approaches were run on Beck for suicidal ideation questionnaire.

Result: In general the difference between Likert scale approach and latent variable approach were higher in 3 question options compared to 5 and also by increasing sample size and correlation between bivariate data, Root Mean Square Errors and bias were decreased. Assuming different probabilities for options, there was a considerably difference between Root Mean Square Errors, bias and standard deviation of estimation of ICC in two models. Using latent variable approach resulted less bias, SD and Root Mean Square Errors especially in lower sample sizes.

Conclusion: Simulations showed when the probability of choosing options of a question are skewed, using this method reduced Root Mean Square Errors especially when the options are less. This method was affected more on standard deviation compare to bias of estimations.

1. Agresti, A. (2013). Categorical data analysis: John Wiley & Sons.
2. Barnhart, H. X., Haber, M. J., & Lin, L. I. (2007). An overview on assessing agreement with continuous measurements. Journal of biopharmaceutical statistics, 17(4), 529-569.
3. Bartko, J. J. (1966). The intraclass correlation coefficient as a measure of reliability. Psychological reports, 19(1), 3-11.
4. Beck, A. T., Brown, G. K., & Steer, R. A. (1997). Psychometric characteristics of the Scale for Suicide Ideation with psychiatric outpatients. Behaviour research and therapy, 35(11), 1039-1046.
5. Biswas, A., & Das, K. (2002). A Bayesian analysis of bivariate ordinal data: Wisconsin epidemiologic study of diabetic retinopathy revisited. Statistics in Medicine, 21(4), 549-559.
6. Chen, C.-C., & Barnhart, H. X. (2008). Comparison of ICC and CCC for assessing agreement for data without and with replications. Computational Statistics & Data Analysis, 53(2), 554-564.
7. Kim, K. (1995). A bivariate cumulative probit regression model for ordered categorical data. Statistics in Medicine, 14(12), 1341-1352.
8. Liljequist D, Elfving B, Skavberg Roaldsen K (2019) Intraclass correlation – A discussion and demonstration of basic features. PLoS ONE 14(7): e0219854McCullagh, P. (1980). Regression models for ordinal data. Journal of the Royal Statistical Society. Series B (Methodological), 109-142.
9. Maxwell, A. E. (1977). Multivariate analysis in behavioural research (pp. 164p-164p). London: Chapman and Hall.
10. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological methods, 1(1), 30
11. Mehta, S., Bastero‐Caballero, R. F., Sun, Y., Zhu, R., Murphy, D. K., Hardas, B., & Koch, G. (2018). Performance of intraclass correlation coefficient (ICC) as a reliability index under various distributions in scale reliability studies. Statistics in medicine.
12. Metrology, J. (2008). Evaluation of measurement data—Guide to the ex pression of uncertainty in measurement. Bureau International des Poids et Mesures.
13. Qin. S., Nelson. L., McLeod. L., Eremenco. S., Coons. S.J. (2019). Quality of Life Research 28, 1029–1033.
14. de Raadt, A., Warrens, M. J., Bosker, R. J., & Kiers, H. A. (2021). A comparison of reliability coefficients for ordinal rating scales. Journal of Classification, 38(3), 519-543.
15. Ren, S., Yang, S., & Lai, S. (2006). Intraclass correlation coefficients and bootstrap methods of hierarchical binary outcomes. Statistics in medicine, 25(20), 3576-3588.
16. shan. G. (2020). Estimation of biascorrected intraclass correlation coefficient for unbalanced clustered studies with continuous outcomes. Communications in Statistics - Simulation and Computation, DOI:10.1080/03 610918.2020.1811332
17. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological bulletin, 86(2), 420.
18. Todem, D., Kim, K., & Lesaffre, E. (2007). Latent‐variable models for longitudinal data with bivariate ordinal outcomes. Statistics in Medicine, 26(5), 1034-1054.
19. Yen, M., & Lo, L.-H. (2002). Examining test-retest reliability: an intra-class correlation approach. Nursing research, 51(1), 59-62.
Files
IssueVol 9 No 1 (2023) QRcode
SectionOriginal Article(s)
DOI https://doi.org/10.18502/jbe.v9i1.13972
Keywords
intra class correlation reliability latent variable multivariate model test-retest
Rights and permissions
Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
How to Cite
1.
Chaibakhsh S, Poorhosseingholi A. A Modification on Intra Class Correlation Estimation for Ordinal Scale Variable Using Latent Variable Model. JBE. 2023;9(1):1-10.