To address these issues, Gwet [9] proposed two new compliance coefficients. The first coefficient can be used with any number of evaluators, but requires a simple categorical scoring system, while the second coefficient, although it can also be used with any number of evaluators, is more appropriate if an ordered categorical scoring system is used. The first correlation coefficient is called a «first-order correspondence coefficient,» or AC1 statistic, which adjusts the overall probability based on the probability that the evaluators will agree on a score even if one or all of them gave a random value. Random evaluation occurs when an evaluator does not know how to classify an object, which can occur if the properties of the object do not match the scoring instructions. Random matching can inflate the overall probability of match, but should not help measure an actual match between assessors. Therefore, as is the case with Kappa statistics, Gwet defined the AC1 tool on random matching, so that CA1 between two or more reviewers is defined as the conditional probability that two randomly selected reviewers agree, since no match is random [9]. Gwet found that kappa gives a slightly higher value than other coefficients when there is a high degree of agreement; However, in the paradoxical situation where kappa is low despite high agreement, Gwet suggested using AC1 as a «paradox-resistant» alternative to the unstable kappa coefficient. Statistically very significant z-tests suggest that we have the null hypothesis that the dimensions are independent (i.e. kappa = 0), and accept the alternative that the match is better than one might expect at random. Do not put too much emphasis on testing Kappa statistics, it makes a lot of assumptions and falls into errors with small numbers. Cohen`s Kappa, Gwets AC1 and percentage of agreement were calculated using agreestat version 2011.3 (Advanced Analytics, Gaithersburg, MD, USA). Weighted kappa partially compensates for a problem with unweighted kappa, namely that it is not adjusted for the degree of disagreement. Disagreements are weighted in decreasing priority from the upper left (origin) corner of the table.
StatsDirect uses the following weight definitions (1 is the default): Cohen J: A match coefficient for nominal scales. Educ Psychol Meas. 1960, 20: 37-46. 10,1177/001316446002000104. For example, the prevalence of depressive PD in the VU-MN pair was 10.53% (2/19 overall), while Cohen`s kappa score was 0.604 (SE.254), AC1 gwets was 0.857 (SE.104), and the degree of agreement was 89%. For the US-SP pair, the prevalence was 12.50% (2/16), Cohen`s kappa was 0.765 (SE .221) and gwets AC1 was 0.915 (SE.087), while the degree of agreement was 94%. Gwets AC1 is the statistic of choice for two evaluators (Gwet, 2008). The Gwet matching coefficient can be used in more contexts than kappa or pi because it does not depend on the assumption of independence between evaluators. McCoul ED, Smith TL, Mace JC, Anand VK, Senior BA, Hwang PH, Stankiewicz JA, Tabaee A: Interrator agreement of nasal endoscopy in patients with a history of endoscopic sinus surgery. Int Forum Allergy Rhinol.
2012, 2: 453-459. 10.1002/alr.21058. Cohen J: Weighted Kappa: Nominal scale agreement provision for staggered disagreements or partial loans. Psychol Bull. 1968, 70: 213-220. If you have only two categories, then Scott`s Pi statistic (with confidence interval constructed using the Thunder-Eliasziw (1992) method) is more reliable than kappa for inter-evaluator matching (Zwick, 1988). Cicchetti DV, Feinstein AR: High approval, but low kappa: II. Solving paradoxes. J. Clin Epidemiol. 1990, 43: 551-558.
10.1016/0895-4356(90)90159-M. Clinicians must ensure that the measures they use are valid, and low reliability among evaluators leads to a lack of trust; For example, in this study, schizoid had a high percentage of match (88% – 100%) in 4 pairs of evaluators; Therefore, high reliability can also be expected between evaluators. However, Cohen`s kappa gave values of .565, .600, .737 and 1,000, while Gwets AC1 returned values of .757, .840, .820 and 1,000, documenting that different levels of agreement can be achieved when these different measurements are applied to the same data set. For example, according to landis and Koch`s criteria, the cohen-kappa value of .565 falls into the «Moderate» category, while gwet`s AC1 value of 0.757 falls into the «Substantial» category (Table 7). A good level of agreement, regardless of the criteria used, is important for clinicians because it promotes confidence in the diagnoses made. This study was conducted on 67 patients (56% men) aged 18 to 67 years with an average SD of 44.13 ± 12.68 years. Nine evaluators (7 psychiatrists, one psychiatrist and one social worker) participated as interviewers, either for the first or second interview, which took place 4 to 6 weeks apart. Interviews were conducted to make a diagnosis of personality disorder (PD) based on the DSM-IV criteria. Cohen`s Kappa and Gwets AC1 were used and the degree of agreement between the assessors was assessed in terms of simple categorical diagnosis (i.e., the presence or absence of a disorder). The data were also compared with a previous analysis to assess the effects of trait prevalence. The probabilities of random matching for Cohen`s kappa (e(K)) and Gwets AC1 (e(γ)) were calculated using the formulas given above and in situations where the marginal number was zero (the evaluators agreed 100%), as for avoidants, addicts, passive-aggressive and paranoid in the TW-SR and NW-SR pairs.
Cohen`s Kappa gave a value of «0» for each of them, while Gwet`s AC1 gave a value of 0.858 for Avoidant and 0.890 for the other three – the closest ones in terms of degree of agreement (Cohen`s Kappa could not be calculated with the SPSS program because at least one variable in each 2-way table, on which the association measures were calculated, was a constant). .