The aim of this study was to explore whether change in verbal memory with time in patients with epilepsy is influenced by performance on tasks assessing verbal cognition or attention/processing speed. Thirty-six patients and twenty-five healthy controls were tested twice with median retest intervals of 4.8 and 3.1 years, respectively. Aspects of verbal memory, verbal cognition, and attention/processing speed were assessed. Decline in one verbal memory variable (Cronholm–Molander Memory Test Paired Associates—Delayed Recall) was the strongest correlate of epilepsy. The second strongest correlate was a decrease in one attention/processing speed variable (Digit Symbol). The relationship between decline in verbal memory and epilepsy was not influenced by the decline in attention/processing speed, and the results did not support the notion that limited mental reserves as reflected in impaired verbal cognition or attention/processing speed can explain the relationship between verbal memory and epilepsy.