The semantic structure of accuracy in eyewitness testimony · Gustafsson PU, Sikström S, Lindholm T

Study 1

The aim of the first study was to investigate the semantic space of correct and incorrect eyewitness statements (see Sarwar et al., 2015). We hypothesized (1) that correct and incorrect statements would occupy different semantic spaces, that is, differ linguistically. Furthermore, we examined the potential content of such a linguistic difference. Our idea was that witnesses might express themselves in a friendlier, warmer style when remembering incorrectly, in order to compensate for a potential feeling of “lesser competence” from providing an incorrect answer. This idea is based on the stereotype content model (Cuddy et al., 2008; see also Abele and Wojciszke, 2007), which suggests that people often evaluate others across two dimensions: communion/warmth (e.g., friendly, kind) and agency/competence (e.g., skillful, intelligent). When people feel that they are underperforming in one dimension, they tend to compensate by emphasizing the other dimension (Holoien and Fiske, 2013; Lindholm and Yzerbyt, 2018). For example, a person that feels low in competence can compensate by appearing friendlier. We expected that this might come into play in an eyewitness context; when witnesses recollect something inaccurately, a feeling of lacking competence should arise. The witness may then express themselves in a “warmer” fashion, and use a more positive tone. We therefore hypothesized (2) that incorrect statements would be more semantically similar to the concept of communion and (3) have a higher valence compared to correct statements. Conversely, recalling correct memories should instill a sense of competence. Witnesses may then express themselves in a more dominant fashion. We therefore also hypothesized (4) that correct statements would be more closely related to a dominant-semantic representation.

Data availability

The data and code for both studies (including the mega analysis) are available at https://osf.io/f83wt/?view_only=5b1ea8cf6d944a7da44fb3f9eb82dc29 and https://osf.io/ztcs8/?view_only=a578b381301541ab9a96b5e444a0eef5.

Method

Dataset

The dataset was originally published in Lindholm et al. (2018) as Study 1, in which 34 participants (Mage = 31.06; SD = 7.37, 100% men) were interviewed in Swedish as witnesses after having watched a mock crime video of a kidnapping. These interviews were videotaped and then transcribed verbatim. The interviews included a free recall phase, immediately followed by cued recall questions (e.g., “how old was the first offender?”). Objectively verifiable statements from answers to the cued recall questions were cataloged and coded for accuracy. The dataset comprised 783 statements (78.16% correct).

The study was conducted in full in accordance with the ethical principles outlined in ALLEA (All European Academies) (2017), and with the ethical principles outlined on http://www.codex.vr.se/, and with the 1964 Helsinki declaration and its later amendments. The studies did not include factors that require ethical vetting according to Swedish legislation on research ethics Svensk författningssamling (2003). All participants had given written informed consent to participate. For a full description of the procedure, see Lindholm et al. (2018).

Data analysis

The cataloged statements were quantified by using a version of the LSA (Landauer, 2007) algorithm as described in Kjell et al. (2019). The semantic space was created from the Swedish version of Google N-gram that is publicly available for download at https://catalog.ldc.upenn.edu/LDC2009T25. In this database we used 5-word grams that are available without further pre-processing. First, a co-occurrence matrix was created, where the columns were the 120,000 most common words, and the rows the 50,000 most common words. Each cell represents the number of times the word in the column and the word in row co-occurs in the 5-word grams. The content of the cells was normalized by taking the logarithm plus one. A data compression algorithm called singular value decomposition (SVD) was then applied to the co-occurrence matrix, with the purpose to maintain as much information in the matrix as possible, in a new matrix that is smaller than the original matrix. This resulted in a semantic space where each of the selected words was described by a vector consisting of 300 dimensions. The length of this vector was normalized to one which is a necessary step to scale the semantic similarity scores between –1 and +1. A semantic representation was created for each eyewitness statement by summing semantic representation in each dimension over all words in the statement. The resulting vector was again normalized to the length of one, by calculating the length of the vector and dividing each dimension with that value. This resulted in a 300-dimensional semantic representation for each statement, each with a length of one.

In addition to creating a semantic space with LSA, statements were also quantified using the Bidirectional Encoder Representations from Transformers model (BERT, Devlin et al., 2018). BERT is a deep neural network language model that unlike LSA generates embedding that handles the grammatical structure of the texts. Thus, BERT acts as a similar, yet more refined method to create semantic spaces. Here we selected the multilingual model “bert-base_multilingual-cased” from Huggingface1, and extracted the representation on the last layer (i.e., layer 12) that consisted of 768 dimensions. The length of this vector was normalized to one.

A prediction model for accuracy was created following the method that is specified in Kjell et al. (2019). We first preprocessed the semantic space by applying the SVD algorithm on the vectors representing the eyewitness statements. This was done separately both for the semantic representations generated by BERT and LSA (to maintain a consistent processing of the two different models). We then trained a model to predict accuracy using multiple linear regression, where we optimized the number of dimensions by trying the first; 1, 2, 3, 5, 7, 10, 14, 19, 26, 35, 46, 61, 80, 105, 137, 179, 234, 305, 397, 488, 768 number of dimensions for the BERT model, and up to 300 for the LSA model. The model was then evaluated by using an 11-fold nested cross-validation procedure, where train and test data always was always separated and where optimization of the hyperparameter number-of-dimensions were conducted in the training dataset (i.e., each fold could have different values) set and then applied on the test dataset. The mean number of used dimensions over the folds were 9.5 (with a standard deviation of 4.2) for the BERT and 9.6 for LSA representation (with a standard deviation of 1.5).

To avoid training on the similar information as tested, the folds were selected by manually classifying each fold into one of 11 different themes based on the statements given in the testimonies (e.g., “clothes,” “weapons”). The themes differed in the mean and standard value of correct classification, which could potentially introduce biases in the machine learning algorithm. For example, it could potentially learn differences in the mean values between themes, rather than learn whether individual statements are correct or not. To avoid this potential problem that otherwise would create artifacts in the predictions, we z-transformed the binary accuracy measures (0 and 1) for fold separately. As the mean and the standard deviation is different for each theme, this z-transformation remaps the binary outcome variable to a normal distribution with a mean of zero and standard deviation of one. Thus, following this z-transformation the machine learning problem became a regression model, and not a binary classification problem.

To allow for comparisons between accuracy and semantic measures of communion, valence and dominance, we created semantic representations for each of these constructs. The communion representation was created using the English “communion” word list from Pietraszkiewicz et al. (2019). This list contains words such as “emotional,” “feelings,” “modest,” but also “religious” and “tempting.” We then created a semantic representation of this list using the same method as for the semantic representation of the statements, with the exception that we use an English LSA representation with 512 dimensions from Kjell et al. (2019). That is, we summed the semantic representation in each dimension for all words in the list and normalized the length of the resulting vector to one by dividing each dimension with length of the vector. This resulted in a 512-dimensional semantic representation of the list with a length of one. Following the recommendation of Boyd et al. (2022), we used Google to translate the original data from the participants from Swedish to English.

For valence and dominance we used a list of words in Swedish that were rated on these constructs, as such word lists were available. The dominance measure (ranging from 1 to 9) was created from a Swedish wordlist (Waldhauser, 2022), where participants rated words (N = 857) for their degree of dominance. Examples of high rated words are “alliance,” “speaker,” and “sadist,” and examples of low rated words are “rabbit,” “belief,” and “toy.” The same predictive model as specified above, with the exception that no z-transformation was conducted (i.e., multiple linear regression, see also Kjell et al., 2019) was trained to predict the ratings from the semantic representation of the words. The ten-percentage leave-out cross-validation procedure generated a significant correlation between predicted and rated dominance, r = 0.27, p < 0.001. This model was applied to the text in the current dataset to predict the dominance measure. The valence measure (ranging from-3 to +3) was created using the same method (r = 0.67, p < 0.001), based on a Swedish word list collected by (Stenberg, 2006, N = 288). Examples of words with high (positive) valence are “wisdom,” “love,” and “trust,” and examples of words with low valence are “cancer,” “torture,” and “murder.” The prediction accuracy of the dominance ratings is lower than for valence, which may be because dominance ratings are more difficult to conduct compared to valence ratings. This idea is supported by the variability of dominance ratings, which, averaged over subjects, are lower than for the valence ratings.

Results and discussion

Accuracy and semantic spaces

We t-tested whether the predicted accuracy differed between correct and incorrect statements (Hypothesis 1). This was done by conducting a t-test comparing the predicted accuracy between correct and incorrect statements in the BERT and LSA-models. The results were statistically significant both with the BERT analysis, t(846) = −4.65, p < 0.001, d = 0.39, MSE = 0.88 (see Table 1), and with the LSA analysis, t(846) = −4.35, p < 0.001, d = 0.37, MSE = 0.90 (see Table 1), indicating that correct and incorrect statements differ in semantic content.

Table. Semantic measures of accuracy. (See the original article for the full table.)

Semantic representations

We next turned to examine if incorrect and correct statements differed in relation to specific semantic representations (Hypotheses 2–4). As the results for the accuracy predictions were similar for the LSA and BERT models, we limit, for the sake of space limitations, these analyses to the LSA model. We first calculated a semantic similarity score for correct and incorrect statements, respectively. This semantic similarity score was calculated as the dot product between the specific semantic representation vector (communion/positive valence/dominance) and the incorrect and correct semantic vector. We then t-tested the difference of these semantic similarity scores. Results are shown in Table 1. In line with hypotheses, results showed that incorrect statements were significantly more closely related to a communion representation, whereas correct statements were significantly more closely related to a dominance representation. Results showed no statistically significant result of accuracy on a positive valence representation.

Taken together, these results support the hypothesis that the semantic space differs for correct and incorrect memories. We thus replicate the finding by Sarwar et al. (2015) with both a new sample, but more importantly, with natural verbal output following an eyewitness interview, rather than written down statements about an event. We also found that both of our analyses of semantic space (BERT, LSA) were able to predict accuracy with statistically significant effects. Moreover, our results showed an effect size of d = 0.37 (LSA), which is more sizable than the correlation coefficient found obtained by Sarwar et al. (2015; r = 0.04). A straightforward explanation for these differences is that oral testimonies allow for a greater variation in ways to express oneself compared to writing, which could involve more cues to accuracy.

Regarding the relation between accuracy and specific semantic representations, results were largely in line with hypotheses, as incorrect statements were more closely related to a communion representation, whereas correct statements were more closely related to a dominance representation. This supports the idea of compensating a negative aspect in the stereotype content model by boosting another (Holoien and Fiske, 2013; Lindholm and Yzerbyt, 2018). Effect sizes were small-to-medium (d = 0.27 and 0.31), which is not surprising, as (a) testimonies are not a typical situation where someone would boast (i.e., express dominance) and as (b) witnesses are unlikely to express incorrect memories they believe to be incorrect, thus limiting the possible feeling of lacking competence, which in turn limits the need to promote communion. The latter explanation might also give some indication as to why no effect was obtained for positive valence.