Gregory D. Stevens, PhD,MHS, Kelly Sadamitsu, MPH, Theresa Sivers-Teixeira, MSPA, PA-C1, Christina Penate, MFT, Jason Karlawish, MD, Bonnie Olsen, PhD
There are few structured methods to assess a client’s decisional ability, and none have been evaluated for use by Adult Protective Services (APS) workers. As part of a larger randomized-controlled trial to test the California APS Interview for Decisional Ability (IDA 3.0-CA), we sought to determine the reliability of the instrument. We assessed the extent to which trained APS personnel reached similar conclusions about a client’s decisional ability (i.e., inter-rater reliability) when presented with a client vignette and a completed IDA 3.0-CA form. First, we developed 12 client vignettes based on common client risks. Consideration was given to diversity in gender, race/ethnicity, socioeconomic status, and geography. Second, five content experts familiar with development of the tool reviewed completed IDA 3.0-CA forms for each case. Based on reported case difficulty and level of agreement between experts, eight cases were selected to formally test the instrument’s reliability. Third, 39 APS personnel, who had trained to use the IDA 3.0-CA, reviewed completed IDA 3.0-CA forms and answered questions about the clients’ decisional ability for two randomly-assigned cases. For each case, we calculated the percent correct and inter-rater reliability (Cohen’s Kappa). The percent correct ranged from 67% to 100%, with an average of 87%. The inter-rater reliability for cases ranged from -0.01 to 1.00, with the average across cases of 0.66 (i.e., substantial inter-rater reliability). The results suggest that APS personnel using the IDA 3.0-CA have a high likelihood of reaching similar conclusions about a client’s decisional ability when provided the same client responses.