Inter-Rater Reliability for Assessing Digital Leadership Situational Judgement Test Linguistic Validation using Cohen Kappa

Abstract

Situational Judgement Test (SJT) item was rarely highlighted, especially the digital leadership aspect of teachers using the SJT. Linguistic validity is always neglectable compared to any other types of validation such as content and construct validity. Linguistic validation can be so impactful because it ensures that the majority of the respondents in a given population can understand the instrument and maintain appropriate reading and comprehension levels. Hence, the aim of this study is to examine the linguistic validity of the SJT for teachers' digital leadership using Cohen Kappa analysis. Cohen Kappa beneficial to analyze the agreement between two linguistic experts. This study used a purposive sampling, and it was conducted online. The experts were contacted through e-mail and online messaging applications. These two experts are language experts called Munsyi Dewan (Bahasa) and Munsyi Muda Bahasa who have more than eight years of experience in Malay Language. The two experts have to score the survey which contains 45 items for the nine constructs of digital leadership. Each construct has five items and the scale used is a dichotomous scale or binary data. The result shows a 97.78 percent agreement between the two experts. This means almost perfect agreement between the two experts. However, both experts made several comments to improve the items in the SJT. The comments on the linguistic aspects include particles, the use of italics, prepositions, passive voice, prefixes, vocabulary, and the use of English terms. This study can produce items on SJT for teachers' digital leadership with good linguistic validation and can be easily understood by the respondents. A follow-up study is planned to examine SJT items’ psychometric properties on teachers’ digital leadership using the Rasch Measurement Model.

 Keywords: Situational Judgement Test, digital leadership, Cohen Kappa, teacher, linguistic validity.