The semantic structure of accuracy in eyewitness testimony · Gustafsson PU, Sikström S, Lindholm T

General discussion

In these studies, we examined if correct and incorrect statements in verbal testimonies differed in linguistic content. Results from both Study 1 and Study 2 supported this idea, as did results from the mega analysis, suggesting that accuracy in eyewitness testimony can be predicted with semantic content. We also examined the relationship between accuracy and semantic representations with the goal to find potential explanations in the type of semantic content that separate correct and incorrect statements (see Table 1). Here results were rather inconsistent across studies, but the mega analysis showed that incorrect statements contained verbal content that was more closely related to semantic representations of communion and abstractness, partly supporting previous studies (e.g., Hashtroudi et al., 1990). We discuss these findings below.

Predicting accuracy with BERT and LSA

In these studies, we attempted to predict accuracy by creating and analyzing semantic spaces wherein correct and incorrect statements can be located. We utilized two methods to do so, BERT and LSA. This double analysis allowed us to get a greater glimpse into the reliability of the findings, as the two methods operate with slightly different methods (for details, see Landauer, 2007; Devlin et al., 2018). Overall, results appeared fairly similar with both methods, with small-to-medium effect sizes (see Table 1).

The relationship between semantic space and accuracy in testimonies have (to the best of our knowledge) only been investigated once before, by Sarwar et al. (2015). Whereas they used written testimonies, we analyzed transcribed verbal testimonies. Our results were overall in line with the findings by Sarwar et al. (2015), that is, that correct and incorrect memories occupy different semantic representations. Moreover, our obtained effect sizes (around d = 0.30) appears more sizable than the correlation coefficient found obtained by Sarwar et al. (2015; r = 0.04), suggesting that oral testimonies are preferable to written testimonies when predicting accuracy from linguistic content. Taken together, these studies suggest that semantic analyses using n-grams can be fruitful for predicting accuracy in eyewitness testimony and thus adds a new avenue of use in forensic psychology, in addition to the previous use in determining authorship of written texts (e.g., Johnson and Wright, 2014; Nini, 2018).

Examining proximity to specific semantic representations

In addition to predicting accuracy with semantic representation, we attempted to identify what this semantic representation could consist of. To do so, we first selected and created specific semantic representations relating to aspects that we expected to be more common in either correct or incorrect statements, and then examined if correct and incorrect statements differed in their proximity to these representations. Although we found several statistically significant effects, they did not replicate across studies (see Table 1). To get the best representation of these data, we also examined these semantic representations in the pooled mega analysis. In this final analysis, we found that incorrect statements were more closely related to semantic representations consisting of communion and abstractness, in accordance with predictions. Surprisingly, we did not obtain any noticeable effects for the other representations (i.e., valence, dominance, tentativeness). One explanation is that these semantic representations do not map well to the expressions used when recalling witnessed events. This idea is somewhat supported by the fairly low dot-product values (i.e., means for correct and incorrect statements; see Table 1).

A motivation for this study was to examine if automatic computational analyses could achieve results comparable to the costlier manual coding. In the mega analysis, we examined the proximity of accuracy to a “tentativeness” representation, but did not find that it was more closely related to incorrect statements, as expected. The same nonsignificant result was found when post-hoc testing this also for Study 1 and 2 separately. Thus, the findings do not match original analyses of these data (Lindholm et al., 2018; Gustafsson et al., 2019), in which coders manually coded expressions of effort, including hedges (“I think,” “maybe”), which they found to be more common in incorrect statements. Thus, it appears that there is a value to manual coding that is not captured in the automatic computational analysis. However, a reservation to this conclusion is that the tentativeness representation was examined in English, using Google translate to convert the testimony statements to English. A fruitful endeavor for future studies is thus to examine English-speaking witnesses.

Word cloud

A word cloud was conducted to examine individual words that were more common in incorrect and correct statements, respectively. Interpreting the results available in Figure 1 suggest that certain words pertaining to hedging (“perhaps”), delays (“…”) and non-filler words (“uh,” “uhm”) were more common in incorrect statements, which corroborate the previous coding of these data (Lindholm et al., 2018; Gustafsson et al., 2019). However, the hedge “think” surprisingly appears to be more common in correct responses. It is also possible that the more common “I” in incorrect statements reflect a self-refence to cognitive operations (“I think I was imagining”/“I never had time to…”), which would be in line with reality monitoring (see Hashtroudi et al., 1990; Stromwall and Granhag, 2005; Kensinger and Schacter, 2006; Sporer and Sharman, 2006). A further interpretation in line with reality-monitoring is the word “saw” (past tense of “see,” not the tool) showing up more often in correct statements, which could suggest more sensory details in correct statements. There also appear to be more conjunction words in incorrect statements (“and,” “or,” “but”), potentially indicating that longer sentences are more likely to be incorrect, perhaps due to containing more (fine-grained) details.

Practical implications

Our study has found evidence that the semantic representation of incorrect statements differs from correct statements. However, a reasonable question is to what extent this is practically useful, as the differences are rather small. We want to stress that—given the current data—the suggested algorithm should not be decisive in discriminating between incorrect or correct statements in important real-life settings, such as court decisions. However, we believe that the proposed methods may provide additional information that might otherwise be overlooked when researching accuracy. For example, the current results suggest that we could potentially get a more complete picture of accuracy by examining the type of information in a statement (e.g., communion, abstractness and tentativeness).

Limitations

Contexts that are different from the ones studied here, or different levels of difficulty of remembering the asked for information, may limit the generalizability of our findings. Furthermore, there are limitations related to ecological validity, where the data used to obtain the testimonies in this study were carried out in a lab environment, in which participants watched a staged crime and were able to keep their full attention on the event. This is not representative of many real-life witnessed events (see Garrett, 2011) and these results can therefore not be directly generalized to real-life eyewitnesses. A clear avenue for future research is to examine the semantic content of statements from real eyewitnesses.

A final limitation is the limited sample used in these studies. Although the total sample in the mega analysis comprised a sample of 2,342 statements, these came from only 66 participants. To establish better reliability of findings, more analyses of witness testimonies are needed.