Semantic relatedness and the efficacy of retrieval practice · Gupta MW, Pan SC, Rickard TC

Introduction

The testing effect (TE)—a phenomenon wherein retrieval-based learning enhances episodic memory more than restudy—has been extensively studied and is well-documented across multiple experimental designs, including variations in test type, material type, feedback condition, retention interval, training sequence, and levels of prior learning1–10. Although the TE is primarily considered an episodic memory phenomenon, there has been substantial interest in the role of semantic memory in the effect11–20.

Only a few studies, however, have directly investigated whether the efficacy of testing relative to a restudy control condition (henceforth, the relative efficacy of testing) varies as a function of semantic relatedness. A meta-analysis of the retrieval practice literature by Rowland (2014) found that the magnitude of the cued recall TE did not significantly differ when the stimulus materials were cue-target pairs that are related (Hedges’ g = 0.66) versus unrelated (g = 0.67). The amount of data available for that analysis, however, was acknowledged to be limited. Further, those effect sizes were estimated in the context of unmatched, between-study differences in relatedness rather than through controlled experiments.

To our knowledge, there are only two contemporary papers (c.f.21,) in which cue-target relatedness was manipulated within experiments. Carpenter (2009) conducted two experiments in which study and training phase materials were word pairs with high or low relatedness. During training, pairs were either restudied or tested using cued-recall without feedback, and the final test involved free-recall of the target word from each pair. In neither experiment was there an effect of relatedness on final test performance in the restudy condition. In Experiment 1, the TE on the final test was smaller for high relatedness pairs, suggesting that the relative efficacy of testing may be weaker in that case. In Experiment 2 there was no relatedness effect on the TE. Walsh and Rissman (2023) compared the TE for low and high relatedness word pairs using cued recall without feedback in both the training and final test phases. In the overall data, they observed no statistically significant difference in the TE magnitude across the two relatedness levels. In a secondary analysis, they observed a smaller TE in the high than in the low relatedness condition when only including test items for which a correct response was made during the training phase. However, the interpretation of that analysis is complicated by the fact that final test proportion correct for the full set of restudied items was compared to final test proportion correct for the subset of test items that were answered correctly on the training phase test.

For current purposes, a limitation of the Walsh and Rissman (2023) and Rowland (2014) studies is that the interactions in proportion correct between training task (restudy vs. testing) and relatedness level are uninterpretable. With the exception of special cases (most notably a crossover interaction), if the quantitative mapping between the underlying psychological processes of interest and the observed dependent variable is not known, then an observed interaction (or lack there-of) in the dependent variable is theoretically uninterpretable. Two prior papers have focused extensively on the issue of uninterpretable interactions22,23. As an illustration of this problem in a memory context different from that of the present work, consider an experiment using word pairs with factors of word type (concrete or abstract) and study repetition (1x or 4x). After the study phase, there is a cued recall test for each pair. Figure 1 depicts a plausible non-linear mapping between the psychological construct of memory strength and the observed proportion correct on the final test. In this example, the difference in memory strength for abstract vs. concrete pairs is larger following 4x study (2.5 units of change) than following 1x study (1 unit). Hence, there is an interaction at the psychological level. However, the non-linear quantitative mapping yields no interaction in the observed proportion correct. Generally, if the research goal is to understand psychological processes (rather than to draw purely applied inference), and if the quantitative mapping function from the psychological construct to the dependent variable is unknown and cannot be modeled successfully for at least the two levels of one of the variables (as in the present work), then the observed interaction in the dependent measure is in most cases not interpretable. The same reasoning holds if there is no interaction at the psychological level but there is an interaction at the level of the dependent measure.

Given the substantial interest in the role of semantic memory in the TE, methods for circumventing the problem of uninterpretable interactions are needed. In the current paper we use two approaches22: (1) application of an empirically supported model for low relatedness materials as a yardstick against which to evaluate the relative efficacy of testing for high relatedness materials, and (2) application of a variant of the matching strategy – in the present case, matching groups on restudy performance and determining whether there are testing effect differences—in the context of cumulative proportion correct distribution analysis. In the next two sections we summarize such a model, the dual-memory model of test-enhanced learning, and the empirical work supporting it. We then describe three hypotheses about the effect of semantic relatedness in the context of that model.

Here we summarize aspects of that model that are central to the current work, drawing on previous work4,9,24. The core claim of the dual-memory framework (Fig. 2) is that, whereas restudy strengthens the memory retrieval route that is formed during initial study (study memory), a test with feedback trial both strengthens study memory and forms a separate memory of the test event (test memory), yielding two independent retrieval routes for the tested items on the final test (through study memory and through test memory). Here and elsewhere, the strength of a memory is positively related to probability of recall of the target (given the cue) through a specified route at some later time.

More specifically, on the first test trial following an initial study trial, the presentation of the cue creates a new episodic memory of that cue in the context of the goal to retrieve the target–what we refer to as cue memory. The target can then be retrieved in one of two ways: from the initially encoded study memory from study phase or from the test memory, which is created through retrieval in the training phase, regardless of retrieval accuracy, provided that correct answer feedback is provided (henceforth, feedback). During test trials with feedback, study memory is also strengthened—either because successful retrieval reinforces the association acquired during the study phase or because feedback supplies the correct target on an initially incorrect trial. The strength of a test memory does not depend on whether the target was correctly retrieved on the test trial, an assumption that is supported by prior work9,25. It is further assumed that incorrect target retrieval in the context of feedback does not affect learning or retention26.

To facilitate development of a quantitative model that is consistent with the framework outlined above and that makes precise and testable predictions, Rickard and Pan (2018) made three simplifying assumptions regarding training phase learning. Assumption 1 is that study memory strengths after training (and on the final test) are identically distributed for restudied and tested-with-feedback items. This assumption is plausible if we stipulate that (1) study memory strengthening occurs whenever the cue and target are simultaneously active in working memory (as during restudy and feedback processing) and (2) the duration of that joint activation is not a critical factor in learning (at least beyond some minimal duration). The assumption is consistent with our findings in prior work of equivalently good dual memory model fits using 1 s and 2 s of feedback. Assumption 2 is that study and test memory strengths are identically distributed. Assumption 3 is that study and test memory strengths for tested items are stochastically independent. These assumptions are made on the basis of simplicity and we do not necessarily expect them to hold in the general case.

Given a few more secondary assumptions, Rickard and Pan (2018) showed that, for a given participant and item, probability correct in the test condition of the final test (P­T) is given by the probabilistic inclusive-or equation for the case of stochastic independence:where PT-s is the probability of correct retrieval through study memory for a randomly selected item in the test condition, and PT-t is the probability of correct retrieval through the test memory for a randomly selected item in the test condition.

As indicated by the foregoing discussion, PT-s is assumed to equal PT-t in that equation. It also follows from the model description above that probability of correct retrieval through study memory in the restudy condition (PR) is the same as the probability of correct retrieval through study memory in the test condition (PT-t). Hence, in this simplest case quantitative instantiation of the theoretical framework, PT-s = PT-t = PR. Equation(1) can thus be reduced to,

and the equation for the TE is,

The model predicts that the probability correct in the test condition (PT) and the TE magnitude are both a function solely of probability correct in the restudy condition, PR. To a close approximation, the predictions of Eqs.(2) and (3) generalize to subject-level proportion correct without modification:

Thus, the predicted TE according to the model the value of PCT-predicted minus PCR:

The model makes PCT and TE predictions independently for each participant and with no free parameters. Although a model with no free parameters surely has boundaries in applicability, it has proven useful as an initial quantitative implementation of the dual-memory framework when applied to the low relatedness materials and the cued-recall TE design that is used in the current paper.

Rickard and Pan (2018) showed that the quantitative model closely predicts proportion correct in the test condition, and hence the TE, across 512 participants from multiple cued recall experiments involving different material types (word pairs, triplets, and history and biology facts) and retention intervals (1, 2, and 7 days). They showed that the model provides a viable account of the TE for the cases of both feedback and no feedback during training.

Cumulative distribution plots show that the model’s predictions hold not only for mean proportion correct but also for the proportion correct distribution (Rickard, 2020). An example cumulative plot and model fit to word-pair data from Pan et al. (2015; Experiments 1 and 2 combined) is shown in Fig. 3. In that plot, PCR is sorted from smallest to largest across all participants. The PCR and PCT values are also independently sorted and plotted from smallest to largest. There are 242 equally spaced quantile values on the x-axis, scaled from zero to one, representing each of the 242 participants. Hence, in this type of plot, a given participant’s PCR and PCT values would occur on the same quantile only by chance. Finally, the PCT-predicted value at each quantile is the Eq. (1) model prediction, based on the PCR value at the same quantile (for further discussion, see Rickard, 2020).

We considered three plausible scenarios for the effect of relatedness that can be framed within the dual-memory model. In scenario 1, the relative efficacy of testing is the same across relatedness levels (low and high). In the dual-memory model, this scenario would correspond to equivalent expected study and test memory strengths over items in both the low and high relatedness group. Overall memory strengths and associated final test proportions correct are expected to be higher in the high relatedness group, but study and test memory strengths would remain equivalent within each group. In this case, the model predictions Eqs.(1) and (2) will hold for both groups. This scenario defines equivalent testing efficacy for low and high relatedness groups in the context of the model.

Because of the theoretically predicted (and empirically established) quadratic relationship between restudy and test proportion correct, Scenario 1 does not imply that there must be equivalent TE magnitudes in the two groups (for this and related reasons, we differentiate in the current work between the TE magnitude as a proportion correct difference score and the relative efficacy of testing). Rather, TE magnitudes in this scenario may be the same in the two groups or may be different, depending solely on PCR values. If, for example, a typical participant in the low relatedness group has PCR = 0.3, then the predicted TE for that participant is 0.21 Eq.(1), and if a typical participant in the high relatedness group has PCR = 0.7, then the predicted TE is also 0.21. In this case, there would be no interaction in an ANOVA on observed proportion correct with factors of relatedness level and training task. Conversely, if a typical participant in the low relatedness group has PCR = 0.5 (yielding a predicted TE of 0.25), and a typical participant in the high relatedness group has PCR = 0.8, (yielding a predicted TE is 0.16), then there would be an ANOVA interaction, even though in the model study and test memory strengths remain equivalent within each group and hence the relative efficacy of testing remains the same. For this reason, an ANOVA interaction test can be misleading and is not a reliable method for assessing whether relatedness affects testing efficacy.

This discussion illustrates the general problem of theoretically uninterpretable interactions22,23, wherein an interaction in the observed dependent measure does not necessarily indicate an interaction in the underlying psychological processes of interest, and conversely, when the absence of an interaction in the dependent measure does not necessarily indicate an absence of interaction at the process level. One preferred approach in this case is to not rely on the ANOVA interaction test, but rather to evaluate fits of an empirically supported theoretical model, such as the dual-memory model as applied here. A good fit of that model to the means of both groups would support Scenario 1, regardless of whether the TE magnitudes for the two groups are the same or different. We also applied a second, complementary strategy for addressing the interaction problem (a variant of the matching strategy22) in which the cumulative proportion correct distributions were used to determine whether the TE magnitudes of each group match the model prediction across all levels of restudy proportion correct.

Scenario 2 is that high semantic relatedness reduces the efficacy of testing. In this case, the model prediction should overestimate the mean TE magnitude in the high relatedness group, and the TEs at all matched levels of PCR in the cumulative distributions should be smaller than predicted. The third scenario is that high semantic relatedness increases the efficacy of testing relative to restudy. In this case, the model prediction should underestimate the mean TE magnitude for the high relatedness group, and the TEs at all matched levels of PCR in the cumulative distributions should be larger than predicted.