Evaluating the Effectiveness of a Memory Aid System

By 2050, more than 20% of the world’s population will be over the age of 65 [1]. Researchers have investigated the impact of age on sensory, motor, and cognitive abilities (for a summary see [2]). For example, older adults exhibit lower memory performance than younger adults [3]. Age also affects prospective memory, a term coined for tasks in which a person has to remember to perform previously planned actions [4, 5, 6]. Prospective memory is essential for independent living [7], thus it is not surprising that there has been increased interest in the effect of aging on prospective memory. Results show that prospective memory performance is worse in older adults suffering from age-related diseases such as dementia [8, 9] or mild cognitive impairment [7], but also in normal aging [10, 11, 12]. In the current study, we aim to evaluate a system to support older or disabled adults in daily bathroom activities on its effectiveness as a prospective memory aid.

Since prospective memory performance declines with age, researchers have started to test potential interventions to assist older adults in remembering daily activities. These interventions include cognitive trainings such as implementing intentions [13] or simple technical devices to remember to take medication [14]. Due to the progress in computer science, ‘assistive technology’ – technology that allows a user to do tasks that would otherwise be unsafe or not possible [15] – has frequently been suggested for compensating for age-related cognitive decline (e.g. [16, 17]). For example, assistive technology can provide people with a broad spectrum of devices intended to facilitate memory tasks in everyday life (for reviews, see [16, 18, 19]). One particular class of assistive technology are ‘context-aware systems’ [20]. Such systems are able to detect and interpret user activities. Based on this information, these systems can, for example, recognize deviations in activities, such as hand washing [21] or daily or weekly routines [22], and issue reminders or prompts for specific tasks or task steps. Context-aware systems seem to be a promising way to assist older adults in daily activities such as bathroom routines. In addition to keeping privacy, these systems can potentially reduce the workload of caregivers who must frequently observe and prompt older adults to do the activities, if necessary.

Although great progress has been made in developing context-aware systems, data on the effectiveness of such systems is rather sparse (for reviews see, [20, 23]). Evaluation is often conducted with small samples and qualitative data, which is appropriate when considering which elements of a program or prototype should be revised. However, it should not mark the end of an evaluation. More extensive studies – preferably randomized controlled trials – should be carried out to evaluate the effectiveness of these systems; however, these evaluations are currently missing [23].

In the current study, we aimed to experimentally evaluate the context-aware system InBad (engl. InBath) on its effectiveness as a prospective-memory aid. InBad is a context-aware bathroom that is part of the Fraunhofer-InHaus-Zentrum ([24]; for a brief English summary, see [25]). InBad includes a touch screen installed in a mirror that enables the user to interact with the system. Conceptually, after learning the user’s bathroom routine, the system can detect deviations from this pattern (i.e. forgotten tasks) and prompt the user by displaying a reminder on the mirror and issuing an acoustic signal. For example, if a user usually brushes his teeth in the morning and forgets to do so one morning, the system will display a reminder. The system is intended for older adults including those suffering from early stages of dementia or mild cognitive impairment, as these groups are more likely to forget activities of daily living like medication or hygiene.

The purpose of the current study was to evaluate the effectiveness of the system as a memory aid. Therefore, the touch screen was not controlled by the system, but by an experimenter in the background. This so called ‘Wizard of Oz’ technique [26] allows the participant to interact with the system without the need for a fully implemented system. To establish a baseline for the effectiveness of such a system with a sufficiently large number of participants, we conducted this first evaluation with healthy older adults at the age of 60 or above instead of cognitively impaired participants. Since the participants were not cognitively impaired and in order to increase the reliability of our measure, we chose to simulate a challenging morning bath routine with 22 activities to be performed.

The successful completion of prospective memory tasks involves a variety of different mental processes such as planning, a prospective component (remembering that something needs to be done), a retrospective component (remembering what needs to be done and when), and compliance (e.g. [27, 28]). The characteristics and demands of prospective memory tasks can therefore differ a lot, in particular when compared to standard laboratory-based prospective memory tasks [29]. The bathroom routine of the current study, for example, required the execution of multiple different but familiar actions (compared to a single and unfamiliar action as usually employed in laboratory tasks); participants were not involved in an engaging cover task, and participants performed the tasks in an environment that is relevant to the prospective memory task.

In order to assess the effectiveness of InBad as a memory aid, participants were instructed to learn a list of usual bathroom activities such as taking medication and washing hands. Participants had to execute these activities with different forms of memory support. Participants in the ‘no support’ group did not receive any support. This condition served as baseline condition. Participants in the ‘list support’ group were able to access a full ‘to-do’ list of all activities they had to execute. In the ‘system support’ group, participants were assisted by prompting reminders on the mirror when they did not execute an activity for 30 s. The ‘system support’ condition thus represents the functionality of the intended context-aware memory aid system InBad. We assessed prospective memory performance by calculating two measures: (1) the number of activities executed before participants were given any support, i.e. when participants either believed they had executed every activity (‘no support’) or when they did not execute another activity for at least 30 s (‘list support’ and ‘system support’), and (2) the total number of activities executed, including those executed after being supported.

Regarding the first measure, the number of activities executed before participants were given any support, there should be no difference between groups if participants encode the activities equally well and if an expectation of support does not lower motivation to try and remember activities on their own.

In relation to the second measure, the total number of activities, we expected participants in the support groups (‘list support’ and ‘system support’) to remember and execute more activities than participants in the ‘no support group’. We further hypothesized that ‘list support’ may be less effective compared to ‘system support’. Prospective memory failures can be due to failure in the prospective component, i.e. forgetting that something had to be done, but also due to failure in the retrospective component, i.e. forgetting what had to be done [29, 30]. A list to support prospective memory tasks can only aid in problems related to the latter component because individuals have to actively choose to use this memory aid. In addition, perceptive and cognitive problems, such as visual acuity deficits, often present in older adults [31] may decrease the effectiveness of such a nonspecific memory aid.

In addition, it is possible that the participants’ performance may depend on their technical affinity. Czaja et al. [32], for example, reported that computer anxiety and beliefs about computer ability have a significant effect on reported computer usage. In order to address whether affinity to technology affected the execution of activities in this study, we administered a questionnaire measuring technical affinity (TA-EG [33]). Finally, to get a preliminary impression of users’ satisfaction with the system, we asked several questions addressing the participants’ opinion about the support system.

Sixty older adults (28 female) with an average age of 69.6 years (SD = 6.4, range: 60–84) took part in the study. Participants were recruited via local newspapers and randomly assigned to one of the three experimental conditions. Most participants (87.7%) declared their health as ‘very good’ or ‘excellent’, and no participant was excluded due to health-related issues. Participants were tested individually and received EUR 10 for participation.

Stimuli

We created a list of 22 activities typical for a bathroom environment such as taking medication, washing hands, or changing the bathroom towel (see Appendix A for the full list).

Apparatus and Evaluation Tool

Items needed for activity execution (e.g. towel or lotion) were placed in the bathroom environment in an expected location (fig. 1). Participants received the instructions and reminders on a computer monitor positioned half a meter in front of them behind a semitransparent mirror allowing participants to see their reflection as well as the monitor.

Technical Affinity

The TA-EG is a German questionnaire consisting of 19 items measuring technical affinity on four ordinal scales: technical enthusiasm, competence in handling technology, negative beliefs about technology, and positive beliefs about technology. These four scales have satisfactory internal reliability with Cronbach’s α coefficients greater than 0.70 [34]. There are medium correlations between the TA-EG and other measures of technological affinity [33]. The questionnaire is thus a suitable tool to measure technological affinity and competence.

User Satisfaction

We administered a questionnaire using 5-point Likert scale questions asking participants how satisfied they were with InBad (table 1).

The participants were welcomed and asked to read and sign the consent form. The experimental session had four phases: (1) familiarization, (2) learning of activities, (3) execution of activities, and (4) postexperimental questionnaires.

First, participants were informed that they would later be asked to perform several bathroom activities. Participants were given 3 min to explore the bathroom environment and, if necessary, were prompted by the experimenter to open all drawers and cupboards. This familiarization was done to ensure that all participants had noticed all items needed for the execution of the activities.

Second, each of the 22 activities was presented separately in the bottom half of the InBad mirror for 6 s each. The presentation order was the same for all participants (see Appendix A). The order of the list was evaluated during pilot testing as a logical order of bathroom activities. Participants were instructed to study these activities for later execution. Participants were instructed to pretend to do the activities in case they did not want to engage in them. After the presentation, participants were informed about the memory support available to them according to their experimental condition.

Third, all participants were instructed to perform the activities. Participants in the ‘list support’ and ‘system support’ groups were offered assistance when they did not perform any activity for 30 s. The ‘list support’ group was given the opportunity to view a list of all activities (in the same order as presented at study) by saying ‘list’, whereas in the ‘system support’ group, the first activity of those not yet executed (following the order in the study list) was automatically shown on the mirror. In both cases, the information was accompanied by an acoustic signal to attract attention. In the ‘list support’ group, participants were free to use the list as often as they wished from then on. In the ‘system support’ group, participants had to wait an additional 30 s for each further prompt. The phase ended when all activities were executed or the participant indicated that all activities were executed. If participants in the ‘no support’ condition indicated that they could not remember any further activities, they were encouraged to take some time to retrieve additional items before the session was ended. Participants were audio- and video-recorded and the duration of the experimental session was recorded.

During the main phase of the study, the experimenter monitored the participant’s behavior directly (positioned approx. 5 m behind the participant) and via video. Using VisualStudio 2010®, a customized evaluation tool was programmed. This tool was used to present instructions to the InBad monitor and to automatically prompt the reminders in the ‘list support’ and ‘system support’ conditions. In addition, the experimenter used the evaluation tool to record and time stamp the participants’ execution of activities.

Fourth, participants were asked to fill in the user-satisfaction questionnaire, the TA-EG, and a demographics questionnaire. The whole session lasted approximately 1 h.

The significance level for all statistical tests was set at α = 0.05.

To assess a possible effect of participants’ expectation to receive memory assistance, we analyzed whether the three groups differed in their performance before participants were given any support. On average, participants remembered to execute 14.0 (SD = 4.3) activities in the ‘no support’ condition, 15.6 (SD = 3.7) activities in the ‘list support’ condition, and 14.8 (SD = 3.3) activities in the ‘system support’ condition. A one-way ANOVA indicated no effect of support condition (F < 1; fig. 2). Thus, the expectation to receive memory assistance did not affect the motivation to execute activities without assistance.

Next, we assessed how much performance improved due to support. To this end, we subtracted the activities executed before being supported from the total number of activities executed. On average, participants executed a total of 15.5 (SD = 3.2) activities in the ‘no support’ condition (i.e. performance after being encouraged to remember further activities on their own), 20.0 (SD = 2.0) activities in the ‘list support’ condition, and 22.0 (SD = 0.0) in the ‘system support’ condition (fig. 2). To examine the effect of providing memory assistance, we calculated the difference between the total number of activities executed and number of activities executed before being supported, for each participant. We compared the three groups via a one-way ANOVA with the difference score as the dependent variable. This ANOVA revealed a significant effect of support condition [F(2,59) = 12.81, p < 0.001, η_p = 0.93]. Bonferroni-corrected post hoc tests showed that participants had significantly higher scores in both the ‘list support’ condition (M = 4.4, SD = 3.65, p < 0.04, d = 0.78) and the ‘system support’ condition (M = 7.25, SD = 3.29, p < 0.001, d = 1.62) compared to the ‘no support’ condition (M = 1.5, SD = 3.8). Scores were significantly lower in the ‘list support’ condition compared to the ‘system support’ condition (p < 0.045, d = 0.82). Thus, both support conditions resulted in better performance compared to no support and, importantly, system support resulted in significantly better performance compared to list support.

Participants in the ‘list support’ condition accessed the list of activities on average 2.6 times (SD = 2.0; range: 0–8) and remembered an additional 4.4 items (SD = 3.6; range: 0–16). Participants in the ‘system support’ condition received a total of 6.3 cues (SD = 3.0; range: 1–12) and remembered an additional 7.3 items (SD = 3.3; range: 1–13). In both conditions, the number of list accesses or system cues correlated with the number of additionally remembered items (list support: r = 0.83, p < 0.001; system support: r = 0.84, p < 0.001).

In order to address whether the different forms of support affected the duration of the experimental sessions, we conducted a one-way ANOVA on duration. There was a significant effect of support condition [F(2,59) = 12.67, p < 0.001]. Post hoc t tests revealed that the ‘no support’ condition (M = 5:29 min, SD = 2:31) was significantly shorter than the ‘list support’ [M = 8:25 min, SD = 2:53, t(38) = –3.278, p < 0.05] and ‘system support’ conditions [M = 9:45 min, SD = 2:49, t(38) = –5.060, p < 0.001]. However, there was no significant difference between the ‘list support’ and ‘system support’ conditions in the amount of time participants needed to complete the experimental session [t(38) = –1.547, p = 0.130].

Technical Affinity and Activity Performance

We calculated Spearman’s rho to correlate the questionnaire data of the TA-EG with the total number of performed activities. None of the four subscales correlated with experimental performance (technical enthusiasm: r_s = 0.20, p = 0.132; competence in handling technology: r_s = 0.06, p = 0.646; negative beliefs about technology: r_s = 0.07, p = 0.454; positive beliefs about technology: r_s = –0.10, p = 0.550). Given α = 0.05 and a sample size of 61 participants, the statistical power to detect medium effects of r = 0.35 was 0.8 (the power analysis was performed with G*POWER3, [35]).

User Satisfaction

We tested for significance by carrying out one-sample t tests against the midpoint of the 5-point Likert scales (3 = neither agree nor disagree) of our user satisfaction questionnaire. By testing against the neutral midpoint, we determined whether the participants agreed or disagreed with a specific item. Table 1 summarizes the results.

The aim of this study was to experimentally evaluate the InBad system on its effectiveness as a prospective memory aid. The results show that the participants using the support remembered to execute significantly more activities compared to both no support and more traditional support via a ‘to-do’ list. Despite improved task performance, participants using InBad did not take any longer to complete the activities compared to participants using the to-do list. Therefore, the InBad system was shown to be the most effective and efficient memory aid.

Importantly, there was no difference between the conditions regarding the number of activities that were remembered before participants were given any support. It is therefore unlikely that participants’ motivation or reliance on the expected memory support affected the results. In addition, there was no association between technical affinity and performance, indicating that affinity to new technology did not influence the results.

In line with previous research, the results show that providing external memory aids can increase prospective memory performance (e.g. [14, 36, 37, 38]). In addition, the results demonstrate a potential benefit of context-aware memory support systems over more traditional memory aids such as to-do lists. Theoretically, this benefit may be explained in two ways. First, context-aware systems initiate specific support. Considering the different components of a prospective memory task, the context-aware system supports the prospective and retrospective component, i.e. it will remind the user that something needs to be done and what it is that needs to be done. A list, on the other hand, still requires users to self-initiate the access of the list and requires them to realize which activities have been performed and which activities still require action. Thus, the list mainly supports the retrospective component and not the prospective component. There is evidence indicating that older adults have particular problems with the prospective component [11], thus the list may have not been consulted regularly enough (average access to list: 2.6 times). Second, when accessing the support list, cognitive problems may have caused problems in the discrimination of items for older adults [31]. Older adults may have had difficulties discriminating those items on the list they had already performed from those they had not performed yet. Discrimination of performed items from items not yet performed is a source-monitoring task, and it is well known that older adults do not do as well as young adults on such tasks [39]. The context-aware support, by contrast, provides relevant and specific reminders. Finally, one might have intuitively expected longer session durations in cases of system support compared to ‘to-do’ list support due to the 30-second delay between prompts in the system support condition. However, we observed no statistically significant difference between groups. This suggests that older adults could not use the list support very efficiently which might have been due to problems in discrimination of items when presented in a single list [31] and additional time demands due to visual search and the demands on memory to check whether a task has been completed or not.

The data on user satisfaction indicates that both list support and system support were overall well perceived, but users did not regard them as beneficial. This may be the case because we investigated healthy older adults who may not perceive a need for support in everyday tasks. In addition, participants might have considered the system support as beneficial if they had experienced the no support condition, as well. Based on the literature, the opinion of the users and in particular the perceived benefits are highly relevant for the adaptation of new technology. Melenhorst et al. [40] found that perception of benefits and lack of perceived usefulness are primary attitudinal factors influencing whether older adults decide to use technology. Therefore, assistive technology must be evaluated in order to guarantee a high benefit to its user. The current study presents a demonstration of the effectiveness and efficiency of a context-aware system.

The study has several limitations. First, the participants were potentially more motivated, independent, or secure about their own cognitive competence compared to the intended users [41]. Given that the system is intended for individuals suffering from cognitive impairments, the effectiveness should be further evaluated with this user group. Second, although participants were familiar with the bathroom activities, the set-up of InBad in the Fraunhofer-InHaus-Zentrum and the specific experimental requirements were novel to the participants. In one’s own personal bathroom environment and bathroom routine, the habitual nature of the task likely improves task performance. Performance in habitual prospective memory tasks depends on associative cuing by preceding actions and environmental cuing [42, 43], which are both more likely in more familiar routines and environments. We considered using participants’ normal bathroom routines. However, since our bathroom environment was unfamiliar to participants, we reasoned that giving all participants the same tasks would result in a more reliable measure and enable better experimental control. By ‘forcing’ participants in all groups to use the same scripted routine, we strengthened the internal validity of the results. Also, remembering one’s own bathroom routine is an easy task for healthy adults. By using a new bathroom routine we created conditions that somewhat mimic cognitive impairment. This is important because ultimately the system is supposed to assist older adults with cognitive impairment. Future research should extend our findings by examining prospective-memory aids using, for example, naturalistic and complex prospective-memory tasks that require participants to plan, retain, initiate, and execute a set of actions [44]. Third, researchers have expressed the possibility that cognitive abilities may decline over time due to an assistive system [45]. We are not aware of any empirical data on the effect of assistive technology on the development of cognitive abilities. In principle, context-aware systems remind only if necessary and therefore the user still performs the majority of tasks without support. However, having the system available might affect how prospective memory tasks are approached [17, 27] and users might start to heavily rely on the system. Fourth, we used the ‘Wizard of Oz’ method to simulate the context-aware system rather than a fully implemented system. The simulated system provided the participants with timely and reliable memory aids. In a fully implemented system, activity recognition and interpretation of specific subtask sequences are probabilistic processes and reminders issued by such a system might not be timely and reliable and potentially cause frustration [21]. Performance with a fully implemented system might thus have been different compared to performance with our simulated system.

Context-aware support systems have been suggested as a promising technology to support older adults in daily activities (e.g. [17, 46]); however, the effectiveness of these systems has seldom been evaluated in controlled experimental studies (e.g. [20, 23]). The present study indicates that a context-aware memory aid supported participants effectively. However, future research is needed to investigate whether these benefits hold with the actually intended users and different task settings. In addition, the effectiveness and usability of a fully implemented system needs to be evaluated. Finally, future studies need to be conducted in the actual home environment to address the effectiveness and potential effects of prolonged system use.

English translation of the original German activities (in order presented):

Stand on scales

Take one pill

Gargle with water

Dilute and use mouthwash

Use cotton bud

Use deodorant

Use perfume

Wash hands

Dry hands

Put dirty towel into laundry basket

Clean glasses with cleaning cloth

Exchange band-aid

Put on watch

Wipe washstand with washcloth

Put washcloth into laundry basket

Water flowers

Refill toilet paper holder

Rearrange bathmat

Spray aroma dish with room scent

Take tissues from washstand

Use lint roller

Use hand lotion

We thank Dennis Rippinger for programming the experimental software and the staff of the Fraunhofer-inHaus-Center for supporting this research.

None.