Memory is the ability of the brain to store information. Given enough stimuli and rehearsal, we can remember data for many years and recall that information whenever we need it. However, not all stimuli are strong enough to generate a memory that can be recalled later. Every day we receive a lot of information through our senses that is lost forever, because it lacked the strength or the repetition needed to generate a lasting memory.

In many situations, that information can be important or necessary: where did I put my keys, what was the name of that man I just met, do I need to buy more milk, etc. As time passes, the aging brain loses information, or is not able anymore to recall some memories. In the extreme situation of Alzheimer's disease, this loss will have devastating consequences for everyday life.

The use of external memory aids to help people to compensate for their memory's deficits is thought to be one of the most valuable and effective ways to reduce the impact of Age-related Memory Impairment (AMI) on everyday functioning [1]. The vision of lifelogging technologies [2] can significantly contribute to realize these external memory aids. Lifelogging technologies sense multiple kinds of data as complete and automatically as possible for storing collections of heterogeneous digital objects that users generate or encounter, including photos, email, web pages and sounds. Although the details are different, overall, these lifelogging approaches propose wearable systems mainly based on still cameras (head-mounted or worn around the neck) [3] or still cameras complemented with sensors such as sound devices [4] or biosensor readings [5]. However, recent works have found that users with collections of thousands of digital photos never access the majority of them [6,7]. These works raise questions about the utility of a digital archive in remembering or reviewing the past.

We believe that the combination of current mobile sensors (such as GPS or RFID readers) and activity recognition techniques [8–11] can enable us to build up a Digital Memory (hereafter Dig-Mem). Whenever a person touches any digitally identified thing, its unique identification and the time and location of the interaction is stored in some persistent digital storage, the Dig-Mem of this person. Furthermore, Dig-Mem structures these interactions into activities, determining if they follow a specific protocol or accomplish a goal. For example, when the user touches a cup, the carton of milk, a spoon and the coffee box in her kitchen in the morning, our approach structures this information as Having Breakfast activity. The recognized activities can be use to trigger autobiographical memory [12] about past events. That is, Dig-Mem could provide the cues to remember past events by means of the associative thinking of the human mind [13].

With Dig-Mem we focus on the part of the natural memory known as declarative memory [14]. Declarative memory (or explicit) is one of two types of long term human memory. It refers to memories which can be consciously recalled such as facts and events. Its counterpart is known as procedural memory (or implicit), which refers to unconscious memories such as skills (e.g., learning to ride a bicycle).

A prerequisite for the realization of the Dig-Mem is the existence of Digital Object Identifiers (DOIs) in the world, as proposed by the Internet of Things paradigm [15,16]. An increasing number of everyday objects (clothing, appliances, books, etc.) are being electronically identified using RFID or NFC tags. This suggests that the DOIs existence prerequisite will be fulfilled in the near future. We expect this issue to fade, but meanwhile, early users have to put the RFID tags themselves. Our proposal of a Dig-Mem will use these DOIs to store information about the interaction with an increasing number of objects, tagged not only by active users of the Internet of Things, but also by manufacturers and vendors, and even by non technical-savvy customers.

The contribution of this work is two-fold: (1) memory-aware services that can be used to remember interactions of persons with things (even when the things are not physically present). These services have been implemented and validated in real environments by students, reaching interesting conclusions about the improvement of catalogs of memory-aware services; (2) A Dig-Mem Browser capable of navigating through those digital memories and represent them thanks to a organization of that memories into activities done with that things, protocols followed using that things and goals accomplished with the manipulation of them. By means of the browser we have performed a case study comparing the evolution of natural memory and Dig-Mem over time. Evidences support that Dig-Mem evolves with time getting close to the user understanding and may exist some kind of synergy between Dig-Mem and natural memory.

From this point, Section 2 presents the overview of our work (memory expressiveness and population techniques). Section 3 presents affinities between interactions. Section 4 describes the aggregation of interactions into activities. Section 5 presents The Memory-Aware Services. Section 6 describes the browser to navigate through Dig-Mem. Section 7 describes the evaluation carried out to validate this work. Section 8 presents a comparison between Natural and Dig-Mem. Section 9 discusses related work. In the final section, we make some conclusions.

This section provides a brief overview of the main concepts of the paper that are going to be developed throughout the next sections.

The core element of our approach, as seen at the bottom of Figure 1, is the Digital Memory concepts. Dig-Mem stores structured information about the interactions of persons with tagged things, and affinities between objects calculated from the interactions. These affinities represent relationships between the objects in the environment. The interactions are grouped into activities, following some rules defined by the user. A user can also define goals or protocols over the activities and interactions.

For example, we can keep track of all the times that the user opens or closes the door of her car (interactions). And we can detect that the user touches the key and the door of her car in the same spatio-temporal frame (affinity). But we can also detect the activity of starting driving the car (interaction with the door followed by interaction with the keys and the handbrake), or even we can check for compliance with the protocol of not drinking before driving.

Core services can be offered to use the information stored on Dig-Mem. These services provide basic information about Dig-Mem elements, as interactions with a specific object, detected relationships with other things or activities where a specific object is involved.

Several core services can be combined or tailored together to provide solutions to domain specific problems or needs. These Domain Specific Services typically are triggered with the user touching a thing of interest and are capable of navigating the Dig-Mem to offer information specific to the domain. For example, by touching the key of the car, the user can ask for the parking location. And holding the hotel key card, the user can ask for a route to the hotel.

Lastly, a Dig-Mem Browser graphically represents the information stored in the Dig-Mem allowing for direct exploration of Dig-Mem in those situations when access to the interactions and affinities stored within are needed.

Interactions comprise the core of the Dig-Mem. But to offer core services, we need to detect the relationships between different things of interest in the environment of the user. Given some criteria that defines a relationship between objects, we will say that two things in our Dig-Mem have affinity if they verify those criteria.

Given the expressiveness of Dig-Mem described in Section 2.1, we can formulate affinity in the following terms:

Two things have spatial affinity when they normally share the same location, even at different times. For example, the relationship that is established between a book and the bookshelf where it is stored.

By means of the interactions and the affinities, we can recognize when something is touched and when two objects share the same space or time frame when are used. However, we need a bigger entity to model more complex actions like driving or having dinner, to do so we define the activities. For constructing our activities we use and Emerging Pattern based recognizer.

Emerging patterns (EP) is a new type of knowledge pattern that describes significant changes (differences or trends) between two classes of data [18]. An EP is a set of items whose frequency changes significantly from one dataset to another. Like other patterns or rules composed of conjunctive combinations of elements, EPs can be easily understood and used directly.

Suppose that a dataset D consists of many instances. An instance contains a set of items i.e., an itemset), where an item is an attribute-value pair. The support of an itemset X, supp_D, is count_D(X)/|D|, where count_D(X) is the number of instances in D containing X. A pattern is frequent if its support is no less than a predefined minimum support threshold. Unlike frequent patterns, EPs are concerned with two classes of data.

We have implemented several services that use the information stored in Digital Memory to test the feasibility of our approach. Core services are built around the dimensions associated with the interactions and with the affinities between them. Domain Specific Services give solutions to domain specific problems by the combination of core services.

At the time of retrieving the information stored in the Dig-Mem, the need of resolved objects gains relevance. We need to ease the task of recognizing an object only by its DOI. We allow the user to directly input a name for a DOI, but there are other techniques to address this problem.

We have developed a technique called Tag sharing for tagged environments, where there are objects shared among different users, for example at home the living room will be the same for all the family. One member of the family could take advantage of the names of objects resolved by the rest of the family. This feature is maximized in environments with a lot of users (more possibilities of finding tagged objects) or where the users share common goals (at work the activities performed with shared objects will similar for more users)

There is an example of this sharing process in the Figure 6. We have a tagged environment shared by two different users (see block 0 of Figure 6).

Each blue circle represents a tagged object. Users start using the Dig-Mem and, after some interactions, affinity relationships are established among the objects (see block 1). Blue triangles represents objects that the user has interacted with, lines represents affinities among objects. We can see that interactions and affinities are not the same for both users. Then (see block 2) User A has declared two activities, indicated by red shapes, involving some of these objects. As part of the activity declaration process he has resolved the names of the objects with its DOI, now these objects are represented by green squares. To do so, the user is aided by an assistant included into the smartphone. The user can resolve a name for a DOI whenever he wants, by means of the browser. Furthermore, when the user interacts several times with a non-resolved object the assistant will ask for a name. User B has declared only one activity and his resolved objects are represented by green pentagons. At this moment the names used by the users for those specific objects are uploaded anonymously to a cloud. The next time that users interact with a non-resolved object, the name used by other users will be proposed. Finally (see block 3) we can see two objects, represented by green pentagons, have been resolved for User A using the names of user B. As for User B, he has three objects, represented by green squares, resolved thanks to the names of User A.

The task of resolving the names of the DOIs is very important to provide a human readable name of an object to the user in the browser. This task should be performed by the user of our approach. In an evaluation environment, the participants know why is important that they perform this task. However in a free living environment, users may not know the importance of the task, or perhaps, they find it as a cumbersome task. Hence, we need to motivate users to resolve the names of the DOIs. There are some works that are focused on the users' motivation necessary for participation [22].

The Dig-Mem Browser features some views so that users can navigate through the memories of his Dig-Mem. For example there is a view to consult the memories of a time frame. The view shows a list of interactions and activities performed in the time frame defined. It can be used as an scheduler that shows all the things done in a day or in a week and has options to hide interactions with non-resolved objects or to show activities that should be performed in the future (because we have defined them as a goal service).

Another example is the most useful objects view. This view enables a list of the most interacted objects, giving them a list of the things that are more commonly used by them. But not only the objects, the user can also consult the list of most performed activities. The view can be configured to calculate this only for a particular period of time, for example a weekend or during the holidays. This can be useful for instance to know the things that are essential for the holidays and carry them on the luggage. To compose this list we just used a core service to find the objects with more interactions in the Digital Memory.

The view traveler objects shows the user a list with the objects that he usually carries on himself. These objects are essential for the user as he always have them wherever he goes, and can be important to the user to be sure that he is carrying all of them. To elaborate this list we search in the Digital Memory for the objects that appear in the maximum number of locations possible.

The services described in this paper have been tested by university staff and third year undergraduate students. The number of people who were involved in the evaluation was twelve. In order to know the experiences (positive and negative) of the participants of the university staff, we did a personal interview at the end of the evaluation. In the case of the students, the evaluations were a mandatory assignment for their Human-Machine Interaction course. They had to do a final paper for this subject describing their usability assessment, and adding their thoughts about the whole activity, with suggestions for new Domain Specific Services and possible improvements.

The system for background feed was built using Open-Source Electronic Arduino prototype platform which reduced considerably the time and expertise needed to build the wristwatch. We used the same smartphone, a Nexus S for foreground feed testing, but in background feed tests its NFC reader was turned off. We put a flag in every Dig-Mem record to distinguish between foreground and background feed, but we used all records for every evaluations.

Figure 7 shows some of our students using the services described in this work:

In the next section we will emphasize some of the findings we have obtained through the analysis of the student behaviour while using the memory-aware services detailed in this article.

We have performed a case study to compare both Dig-Mem and natural memory evolution throughout time. In particular we want to evaluate the ease of remembering things performed in a regular day.

We recorded the case study to compare if the memories that we are going to collect are correct or not. Because of that we used a closed area, a laboratory, place of work of some internship from Universidad San Jorge. We decided to use this environment due to the kind of interactions that take place there. In a classroom the students mainly listen to the lecturer and there is much less interaction with the environment, by contrast in a laboratory of interns the tasks performed are linked to the objects present in that environment, and involve less listening and more object manipulation.

We have chosen interns for the case study so they are familiar with the environment. We choose four interns, all of them are part of a research group. The group of interns is not related to the Dig-Mem project, they don't know about the device before the case study, but they have technical background, avoiding problems with the operation of the device.

The case study starts explaining the interns the use of the device; we give one device for each intern and tag the laboratory with their help. We encourage the interns to tag also personal objects used at the laboratory as their wallets or cellphones. We place a camera and tell the interns that we are going to record the case study, but we don't specify the objective of the case study, to avoid extra efforts by the interns to remember specific things. They know they are going to participate in a case study, they also know they are going to be recorded by a camera and which objects are tagged, but they don't know that the goal is to compare the Dig-Mem and natural memory.

The second day recorded by the camera (The first day interns were excited about the case study and the camera) will be the reference day (hereinafter ref-day), and will be used to evaluate their memories as time passes. It is important to notice that the Dig-Mem will be then locked to start after that day, by this we avoid that the interns remember the things done that day consulting the Dig-Mem.

At the end of the day we met the interns and told them that we need to evaluate their memory. We want to minimize the effect of our questions over their natural memory, so instead of asking directly about if some actions where done or not, we ask them to tell us everything that they have done during the ref-day. We emphasized the importance of details and ask them to take as much time as they need because the intern that remembers more correct things will win a prize. We don't give them feedback, so they don't know how well have performed the test.

We perform this activity three times more, following the same pattern. It is important that the interns don't know if there is going to be another meeting, so they don't make extra efforts trying to remember the actions of the day (or even cheating by writing things down).

After the last meeting we conclude the phase of data collection of the case study. The Figure 8 is a representation of a subset of the data collected from the devices and the written stories written by the interns. We only show the information of one intern to simplify the image, but the results where similar for all the interns.

In the first meeting, the Dig-Mem is just a set of interactions. There are no objects resolved so we can't figure which element represents each DOI. These memories can't be interpreted only looking at the Browser (although we can use them trough core services). By contrast, natural memory has plenty of details. We can watch how the intern remembers that he has watered the plants, but also details as the number of times they filled the watering can or why they decided to perform the action. With the coffee we observe the same; it offers plenty of details, such as the number of spoons and even personal assessments about the action.

The first row shows the information extracted from the Dig-Mem in the meetings. The second row shows the information extracted from the written stories of the interns, there is shown only the data relevant to the activities we are showing, not the full report. Each of the columns shows one of the meetings results, the first meeting is most left.

For the second meeting, the interns have started using the Dig-Mem and they have resolved some objects; when we consult the memories of the ref-day they make more sense, as we can confirm some objects that the intern interacted with. We don't know exactly what happened but is sure that the intern touched the coffee machine or the water tap. By contrast the memories of the natural memory are less detailed than in the first meeting. These activities are still remembered, but some details are forgotten. In addition, the times are fuzzier.

For the third meeting we observe a big change in the two memories. The interns have defined activities of interest for them, when we browse the memories of the ref-day we observe how the activities have changed the interpretation of the memories. We observe that the intern watered the plants and prepared coffee. By contrast the natural memory has been reduced to just the activity; there are no details of them. In addition the interns started to mix memories of different days. They attributed things done other days in that week (we check that with the record) to the ref-day, in the image we can see a representation of that.

For the last meeting we observe that the interns have started to use the Digital Memory to control their activities. For instance they have defined a goal service over the activity watering plants to be sure of performing the activity two times a week. When we observe the memories of the ref-day, the activity watering plants is interpreted as the first of the two planned for the week. By contrast the natural memory has lost even more information, and some of the mixed memories are repeated. We can observe how some of the tasks are even totally forgotten, like preparing coffee.

In summary, the interpretation of the Dig-Mem has evolved over the time, making the memories recorded in the ref-day understandable and giving them some value for the user as time passes. The hours of the Dig-Mem are very accurate. By contrast the natural memory was plenty of details at the beginning, giving much more information than the Dig-Mem (even the one of the last meeting). As time passes this natural memories have suffer from degradation, up to the point of mixing memories of different days or totally forgetting whole activities.

At the end of the case study we unlocked the data of the ref-day of the Digital memories (that has been locked throughout all the case study) and then we showed the browser in that day to the interns. There were different reactions when they saw that information.

Dig-Mem worked as a stimulus in some cases. When users saw the Dig-Mem, they recall the moment in their memories and contribute new details to that activity.

It turns out that digital memories not only evolve getting close to the user perception, they also feedback the natural memory. We are currently planning an specific case study to study the influence of the Dig-Mem over the natural memory and detect if there is any kind of synergy when we use both digital and natural memory.

Several groups [24,25] have recently proposed to incorporate humans carrying smart phones in a sensing data collecting loop. Such a novel approach is shortly called Participatory Sensing. In participatory sensing, a large number of users carrying smart phones contribute to monitoring the environments with their sensing measurements (for instance, Mobile Millennium [26], Simple Context [27], Urban Atmosphere [28]). That is, users cooperate to sense data that enables services of interest for the collective of users. For example, Participatory Sensing enables generic services such as traffic information [29] or pollution [30] status in their city. However, currently existing deployments have suffered from insufficient participants because participants who voluntarily submit their sensing data found no interest to remain actively in the system without being rewarded [22].

Conversely, memory-aware services require users to individually sense data in order to achieve personal and important services for each individual. In fact, memory-aware services are so personal that if two users switch their digital memories then the resulting services become useless. According to the experimentation with our students, the return of investment (battery power consumption and wearing the wristwatch) is greater than their expectations because they could not find services to complement their lack of memory elsewhere.

Significant amount of research and development is being directed also on monitoring activities of daily living of senior citizens who live alone as well as those affected with certain disorders such as Alzheimer's. This field of research is known as Embedded Assessment Technology [31]. Sensors embedded in the home can monitor how older adults interact with objects around the home and can provide objective accounts of behaviors to support self-awareness. Hayes et al. [32] monitor the pill-taking task, they developed a smart pillbox that could monitor when a door was opened and how the box was manipulated. They augmented an existing off-the-shelf pillbox with snap action switches to know which doors were open. Lee et al. [33] extended Hayes' work by introducing more sensors in the home environment in order to run experimentation that validates the benefits of embedded assessment data. Gartenberg et al. [34] describe a mobile health application that collects data relevant to the treatment of insomnia and other sleep-related problems. The application is based on the principles from neuroergonomics, which emphasizes assessment of the brain's alertness system in everyday, naturalistic environments, and ubiquitous computing. However, the focus of the former works is on monitoring rather than compensation as our Dig-Mem does.

On the other hand, the Instrumented Environments capture as many kinds of data as possible through sensors or networks, as in MIT's PlaceLab (http://architecture.mit.edu/house_n/placelab.html). There are also other works that perform this type of capture. Liu et al. [35] and Nemmaluri et al. [36] use a RDIF localization technique that uses steerable antennas to “sweep” a room, discovering, localizing and indexing tagged objects. None of these works say anything about using their approach to improve memories.

Lifelogging Technologies [37] have the potential to provide memory cues for people who struggle with episodic memory impairment. In particular Lee et al. [4] propose to equip memory-problem patients with a device for taking random pictures daily. Then, such pictures are used to improve memory skills by questioning the patient to remember about what he was doing at those moments. Doherty et al. [38] share their experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. These approaches are not focused on directly compensating effects of memory loss. Instead, the proposal intends to assist the patient in a treatment for rehabilitating his memory.

Finally, Specific Location Services [39,40] distribute an object search query to a subset of mobile users that are likely to find a given item. That is, object owners can search for their objects using the infrastructure of mobile phones carried by other users. The successful of this approach relies on environment conditions such as the participant density and their mobility. In our work, the location services (which are based on the previous interactions of the user) are just part of the core services that we use to provide the domain specific services. Our work does not focus on location services; our main goal is to build up a Dig-Mem that provides memory-aware services. These services are influenced by the current dimensions of the memory (DOI, time and space) but by means of experimentation, we are introducing more dimensions to enrich the Dig-Mem and therefore both the memory-aware services and the Dig-Mem browser.

With more and more RFID tags being added to our surroundings Memory-Aware Services can play an important role in our lives providing us an important complement to our fading memory. Specifically, Memory-Aware Services can answer generic questions such as where are my keys and even more complex problems such as if I have followed correctly my medicine prescriptions or if I have watered the plants enough times this week.

This paper shows how Dig-Mem provide services based on our past interactions with things. We can trigger these services either by touching a physical thing (and digitally remembering our interactions or activities with this thing) or even without having access to the physical object by means of its affinity relationships with other things that we can touch.

In this work, we have developed a device to support the approach enabling the user for populating the Dig-Mem. The digital memories are stored in terms of interactions, affinities, activities, goals and protocols. We have also developed a Dig-Mem Browser to enable users themselves to navigate through their digital memories. Furthermore, this Dig-Mem Browser also takes advantage of the developed Tag Sharing technique to speed up DOIs resolution.

This approach has been successfully validated in different environments at a Technological Park by our students. Besides, we have obtained some interesting findings about the capability of a technological user to take advantage of the Dig-Mem and create a tagged environment.

We have observed the evolution of the Dig-Mem over the time noticing a improvement in the presentation of the memories as time passes and the device is used, compared to natural memories, where the level of detail decreases over time. This has lead us to possible synergies between digital and natural memories.