Inter-Rater Reliability of the California APS Interview for Decisional Abilities (IDA 3.0-CA)


Gregory D. Stevens, PhD,MHS, Kelly Sadamitsu, MPH, Theresa Sivers-Teixeira, MSPA, PA-C1, Christina Penate, MFT, Jason Karlawish, MD, Bonnie Olsen, PhD


There are few structured methods to assess a client’s decisional ability, and none have been evaluated for use by Adult Protective Services (APS) workers. As part of a larger randomized-controlled trial to test the California APS Interview for Decisional Ability (IDA 3.0-CA), we sought to determine the reliability of the instrument. We assessed the extent to which trained APS personnel reached similar conclusions about a client’s decisional ability (i.e., inter-rater reliability) when presented with a client vignette and a completed IDA 3.0-CA form. First, we developed 12 client vignettes based on common client risks. Consideration was given to diversity in gender, race/ethnicity, socioeconomic status, and geography. Second, five content experts familiar with development of the tool reviewed completed IDA 3.0-CA forms for each case. Based on reported case difficulty and level of agreement between experts, eight cases were selected to formally test the instrument’s reliability. Third, 39 APS personnel, who had trained to use the IDA 3.0-CA, reviewed completed IDA 3.0-CA forms and answered questions about the clients’ decisional ability for two randomly-assigned cases. For each case, we calculated the percent correct and inter-rater reliability (Cohen’s Kappa). The percent correct ranged from 67% to 100%, with an average of 87%. The inter-rater reliability for cases ranged from -0.01 to 1.00, with the average across cases of 0.66 (i.e., substantial inter-rater reliability). The results suggest that APS personnel using the IDA 3.0-CA have a high likelihood of reaching similar conclusions about a client’s decisional ability when provided the same client responses.

Read More.

Report Abuse Now



Make A Donation



Special Thanks to Judith D. Tamkin

We are sincerely appreciative to Judith D. Tamkin for her gift to help establish the USC Center on Elder Mistreatment’s website. Her deep and personal commitment to eradicating elder abuse is helping to reshape our understanding of elder abuse and ultimately save innumerable older adults from abuse and neglect.