The semantic structure of accuracy in eyewitness testimony · Gustafsson PU, Sikström S, Lindholm T

Introduction

Language enables us to share information and recount events that we have experienced. This ability is crucial in courtrooms, where eyewitnesses commonly testify to what they have seen during criminal events. However, witnesses can lie or misremember, which makes it important to evaluate the accuracy of a testimony. Given the centrality of language in communication, it is not surprising that researchers have attempted to find verbal cues to identify if a witness is lying (see DePaulo et al., 2003), or to tie a suspect to a threat letter (Nini, 2018). In this study, we analyze the verbal content of testimonies from eyewitnesses and examine how correct and incorrect statements might differ semantically.

Predicting eyewitness accuracy with semantic content

Although much research on accuracy in eyewitness testimonies concerns deception detection (for meta-analyses, see DePaulo et al., 2003; Sporer and Schwandt, 2006; Vrij et al., 2017), another important area of research is predicting accuracy in eyewitnesses, as people can remember incorrectly. A common method to evaluate eyewitness accuracy is to ask witnesses to describe how certain they are in the accuracy of their memory, that is, make a confidence judgment. Overall, much evidence suggest that confidence can be a good indicator of accuracy, such that high confidence is given to correct answers (e.g., Juslin et al., 1996; Mickes et al., 2017; Wixted and Wells, 2017; but see also Wade et al., 2018). There are however also other ways to evaluate eyewitness accuracy, such as by directly analyzing the content of the testimony itself. In theory, methods utilizing such an approach should have the possibility to generate more accurate predictions than confidence judgments, as a testimony is a more direct output of memory, and should therefore contain less error. That is, confidence is a metacognitive judgment and as such, is based not just on appraisals of the memory, but also on knowledge and beliefs (e.g., Flavell, 1979; Mueller et al., 2013). In contrast, a testimony is a direct verbal report of memory contents, and is therefore closer to the source (i.e., a correct or incorrect memory). So far, studies analyzing testimony content have evaluated the testimony’s semantic content. Here follows a summary of two such methods.

The first method to predict accuracy in eyewitness testimony from semantic content is based on the reality-monitoring framework (Johnson and Raye, 1981). With this method, researchers analyze the types of details that are expressed in the testimony. Specifically, the idea is that a real (correct) memory will contain more sensory, spatial and temporal details compared to an imagined (incorrect) memory, which will instead contain more references to cognitive operations (e.g., how one came to remember the detail). For example, participants in an experiment by Schooler et al. (1986) watched a presentation that involved a car at an intersection. Half of the participants saw a yield sign at the intersection, whereas the other half of the participants only had the sign suggested to them in a later questioning. When the participants later had to describe objects in the presentation, those who had seen the yield sign (i.e., gave a correct recall) used twice as many sensory details to describe it, whereas those who only had the sign suggested to them (i.e., gave an incorrect recall) used 10 times as many indications of cognitive operations. Other similar findings support the idea that sensory details are more abundant in correct memories and that references to cognitive operations are more abundant in incorrect memories (Hashtroudi et al., 1990; Stromwall and Granhag, 2005; Kensinger and Schacter, 2006; Sporer and Sharman, 2006; c.f. Clark-Foos et al., 2015). However, a drawback is that this method requires an assessment of the overall testimony credibility, and therefore cannot be used to evaluate the accuracy of individual statements within the testimony.

A more recent method predicts accuracy in eyewitness testimonies based on the amount of expressed effort in retrieving a memory. Early on, Smith and Clark (1993) evaluated answers to general knowledge questions and found that incorrect answers more often contained filler expressions such as “uh,” “oh,” and hedges such as “I do not know,” “I think,” compared to correct answers. Later on, studies examining effort in eyewitness testimonies replicated these findings (Paulo et al., 2015, 2019; Lindholm et al., 2018; Gustafsson et al., 2019, 2022; see also Seale-Carlisle et al., 2022). Moreover, these studies corroborate findings that correct memories are retrieved quicker—that is, easier—than incorrect memories (Robinson et al., 1997; Brewer et al., 2006; Koriat and Ackerman, 2010; Ackerman and Koriat, 2011).

Despite the relative success of the reality-monitoring method and the retrieval-effort method, they are both limited by requiring manual coding; coders must manually process each statement and evaluate the respective semantic cues, which can result in mistakes such as missed cues or incorrect coding. Although this shortcoming is largely mitigated by using coder protocols and interrater reliability tests, it remains a costly and lengthy operation. A possible remedy would therefore be to run eyewitness testimonies through a computerized semantic text analysis.

Data-driven analyses of semantic content

Data-driven text analyses are common in psycholinguistic research. In the field of authorship attribution—that is, the evaluation of whether two or more texts belong to the same author—a central idea is that people have a “written fingerprint” that can be detected by a person’s writing style (i.e., stylometry, Coulthard et al., 2016). To find this “written fingerprint,” researchers analyze the occurrence of various features of a text, such as lexical, syntactic, or semantic features, or some combination of them (see Stamatatos, 2009 for an overview). This is mainly carried out with machine-learning methods (e.g., Zheng et al., 2006; Zlatkova et al., 2018). One such method is based on identifying the frequency of co-occurring letters or words, called n-grams (e.g., Kešelj et al., 2003; Koppel et al., 2009; Johnson and Wright, 2014; Kestemont et al., 2018; Nini, 2018). N-grams are sequential items found in a verbal output. For example, the sentence “He was wearing a blue jacket” with a word 3-gram would result in four sequences: “He was wearing,” “was wearing a,” “wearing a blue, “a blue jacket.” By calculating n-grams over entire texts, you get a library of words (or letters) that commonly occur together. This library can then be compared with libraries constructed from the works of potential authors to examine the overlap. For example, Nini (2018) investigated the authorship of letters that were sent in relation the infamous Whitechapel murders in nineteenth century England. Nini (2018) examined over two hundred letters supposedly sent by “Jack the Ripper” and used word 2-grams to find that the two most famous letters likely came from the same person. Besides examining overlaps, n-grams can also be used—as is the case for the current study—to examine differences between texts.

Initial support for a word gram approach to differentiate correct and incorrect eyewitness statements comes from Sarwar et al. (2015), who utilized word 5-grams together with a method called latent semantic analysis (LSA, Landauer, 2007). The LSA analyses a text and creates a “semantic space” that describes the relation between the words in that text. Words that occur more frequently together in a text (e.g., “cute” and “kitten”) are said to be more closely related in this semantic space, compared to words that occur less frequently together (e.g., “cute” and “capillary”). With this method, Sarwar et al. (2015) examined if words used to describe correct statements belonged in a different semantic space compared to words used to describe incorrect statements. They found this to be the case, thus strengthening the idea that we linguistically express ourselves differently when we remember correctly compared to incorrectly.

Despite the positive results, the study by Sarwar et al. (2015) was limited in two important ways. First, the participants wrote down their memories instead of verbally recalling them. This constrains the ecological validity, as eyewitnesses tend to recall experienced events verbally. Importantly, verbal recall likely contains more genuine, unfiltered expressions than written recall, which could allow for greater semantic differences between correct and incorrect statements. Second, the LSA analysis that showed a difference in semantic space between correct and incorrect statements provided no clue as to what made up this difference. That is, it is unknown if the semantic difference between correct and incorrect statements related to the use of function verbs, expressed emotions, or some other semantic context. Sarwar et al. (2015) commendably attenuated this shortcoming by presenting a list of the most frequent words in incorrect statements. Nonetheless, it is difficult to interpret overarching trends. On one hand, certain words could be categorized as hedges (e.g., “could,” “possibly”), which corroborates previous results (Smith and Clark, 1993; Lindholm et al., 2018; Gustafsson et al., 2019, 2022). On the other hand, many words were general nouns that likely refer to specific details in the mock crime video (e.g. “jacket,” “skirt”) and therefore probably do not generalize well.

In the current study, we aim to replicate the results in Sarwar et al. (2015) by examining the accuracy of statements in eyewitness testimonies with a data-driven method focused on linguistic content, and aim to improve on previous limitations by examining transcripts of eyewitness interviews instead of written eyewitness statements. Furthermore, we will examine specific semantic representations to try to decipher the content of the previously found semantic difference between correct and incorrect statements. Finally, we also analyze data both with LSA, and another natural-language processing technique (“Bidirectional Encoder Representations from Transformers model” [BERT] Devlin et al., 2018) to evaluate the reliability of findings.