Mega analysis
Next, we decided to pool the data from Study 1 and 2 and make a mega analysis (Eisenhauer, 2021) out of both datasets (N = 2,342, accuracy rate = 75.79% correct). Such an analysis should give a more representative picture of the relation between semantic space and eyewitness accuracy due to the larger sample. Furthermore, the larger dataset should allow for better predictions, as the machine-learning algorithm (as specified in Kjell et al., 2019) can learn from more data.
We conducted the same tests as in the first two studies, that is, examined the semantic spaces of correct and incorrect statements with BERT and LSA, as well as examined potential differences in specific semantic representations (communion, dominance, positive valence, abstractness). Additionally, we decided to test if correct and incorrect statements differed in relation to a tentativeness semantic representation. The idea for this is straightforward: in the studies from which these datasets originated (Lindholm et al., 2018; Gustafsson et al., 2019), results showed that incorrect statements were expressed with more indications of effort (“effort cues”) than correct statements. One of these types of effort cues was hedges, that is, hesitations and commitment avoidance such as “I think,” “perhaps.” These results were based on effort cues that had been manually coded. In this study, we wanted to examine if these results could also be obtained without manual coding, that is, by data-driven analyses. As a proxy for effort/hedging, we used the “tentativeness” word list from the LIWC dictionary (Pennebaker et al., 2001), and applied the same method for measuring this as for the “communion” word list. This list contains words such as “guess,” “possibly” and “hesitate,” but also “darken” and “mysterious.”
To examine differences in semantic space, we again calculated t-tests with the BERT and LSA-models. The results were statistically significant both with the BERT analysis, t(2383) = −5.78, p < 0.001, d = 0.28, MSE = 1.01 (see Table 1), and with the LSA analysis, t(2383) = −9.50, p < 0.001, d = 0.45, MSE = 1.00 (see Table 1).
We then tested differences in semantic representations between correct and incorrect statements. Results are shown in Table 1, and showed that incorrect statements were significantly more closely related to the communion representation, and the abstract representation. The remaining representations were not statistically significant, including the newly tested tentativeness representation.
Post-hoc analyses
Given that we discovered new potential semantic representations as the research progressed, we decided to examine these predictors post-hoc also for the individual data sets. We first examined the data in Study 1, and explored whether incorrect and correct statements differed in abstractness. Results showed no significant results (see Table 1). We next examined the tentativeness representation, and again found no significant results. Moving on to Study 2, a post-hoc exploratory examination also showed no significant effect of tentativeness.
Keyword analysis
Finally, we wanted to explore if we could single out specific words that discriminate between correct and incorrect statements. This was done by analyzing combined datasets (i.e., the data in the mega analysis) using Chi-square tests testing whether the frequency of each unique word was more common in the correct statements (plotted to the right) compared to incorrect statements (plotted to the left; Figure 1). Words that were significantly different following Bonferroni correction for multiple comparisons were included. The analysis showed that incorrect statements were uttered more commonly with high frequency short function words (e.g., “a,” “to,” “the”), words related to uncertainty, ambiguity, or doubt (e.g., “perhaps,” “also,” “mm,” “eh,” “ehm”) and pronoun coding for the participants (“I”). Correct statements were instead uttered more commonly with pronouns related to third persons (“he” and “she”).
Figure. Word cloud analysis of words that are significantly more frequent in incorrect statements (left side) and correct statements (right side). (See the original article for the figure.)
CC BY 4.0. "The semantic structure of accuracy in eyewitness testimony" by Gustafsson PU, Sikström S, Lindholm T, Frontiers in psychology (2024). https://doi.org/10.3389/fpsyg.2024.1211987. Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Peer-reviewed and open access; full text from PubMed Central (PMC11040688). Headings and layout adapted for reading; figures and tables are in the original.