Do We Understand the Relationship between Affective Computing, Emotion and Context-Awareness?

Abstract

Historically, the utilization of context, the range and scope of context-aware systems, and the levels of computational intelligence in such systems have been very limited. While the inherent complexity is a significant factor, a principal reason for these limitations lies in the failure to incorporate the emotional component. Affective computing technologies are designed to implement innate emotional capabilities and the capability to simulate emotions and empathy; thus, intelligent context-aware systems with affective computing provide a basis upon which we may effectively enable the emotional component. Moreover, machine cognition relies upon affective computing technologies to provide a basis upon which the emotional component may be incorporated. This paper poses the question: do we understand the relationship between affective computing, emotion and context-awareness? The conclusion drawn is that while affective computing and the need for the incorporation of the emotional component is generally understood and domain-specific strategies to enable implementation have been proposed, there remain important challenges and open research questions in relation to the cognitive modelling and the effective incorporation of affective computing and the emotional component in intelligent context-aware systems.

1. Introduction

Context-Aware Systems (CAS) implement context as it applies to entities, an entity being defined as a: “person, place or physical or computational object” [1]. This definition clearly identifies an entity as an ‘individual’, ‘computational device’, or ‘computational object’. This is relevant to this paper as consideration is given to context-awareness as it applies to humans and intelligent agents (which include humanoid and software robotic systems).

CAS [2] are generally designed to provide personalised service provision in a broad and diverse range of domains and systems including for example: ‘recommender systems’, ‘collaborative computing’, ‘healthcare systems’, ‘pedagogic systems’, and ‘financial systems’ [2,3,4,5,6]. Current approaches to enable personalisation are generally implemented in pervasive CAS and function on the basis of an entities context (also termed a situated role) [2]. A context describes a users prevailing ‘personal’, ‘environmental’, ‘proximate’, and ‘social’ situation [7] and is: highly domain and application specific, potentially highly dynamic, and must reflect a users prevailing dynamically changing state [2]. The issue historically for CAS has been the relatively limited use of the available contextual information (CI); the general usage being limited to spatio-temporal, identity, and possibly proximate information [7]. A broad and diverse range of context factors combine to form a context definition [2,7,8,9]; in fact, almost any information available at the time of an interaction that can be codified and digitised can be considered to be contextual information [2].

Currently, research is attempting to extend the range and scope of CI utilized using multi-modal and multi-factorial approaches; the issues lie the inherent complexity of context [10] and the processing of contextual information to provide decision-support whilst accommodating Constraint Satisfaction and preference Compliance (CS) [11]. In considering the implementation of the emotional component, while in the laboratory cognitive data has been effectively captured and processed, an important factor for ‘real-world’ applications and systems lies in the development of non-invasive sensors to enable the capture of cognitive data [12,13]. The restricted levels of computational intelligence are a common feature of CAS, to address this the emotional component must be incorporated in context definitions to enable the identification of an entities emotional reaction to stimuli. Moreover, machine cognition and machine consciousness will be reliant upon affective computing technologies (AC) with the effective incorporation of the emotional component to enable emotive and empathetic responses in intelligent agents and systems.

This paper poses the question: do we understand the relationship between affective computing, emotion, and context-awareness?. The conclusions are that: (i) the relationship of emotion and AC to context and CAS is generally understood, and (ii) domain specific implementation strategies to address the generally fuzzy nature of context have been developed. However, there remain significant non-trivial challenges and open research questions in relation to the cognitive modelling that is required to effectively implement emotive response to stimuli in CAS.

The remainder of this paper is structured as follows: emotion is introduced in Section 2 with context-awareness and AC considered in Section 3. An overview of research into the human brain is presented in Section 4. Cognitive science with cognition and machine cognition is considered in Section 4.1 and Section 4.2 respectively. Context modelling is introduced in Section 5 with the proposed implementation strategy presented in Section 6 with a brief overview of alternative methods considered set out in Section 6.1. The paper concludes with a discussion in Section 7 where challenges in implementing AC in CAS are considered with open research questions and directions for future research. Concluding observations are presented in Section 8.

2. The Emotional Component

Emotion and its relationship to machine cognition has been a fertile field of research over many decades with research investigating human responses as they relate to the cognitive and psychological aspects of the subject, for example see [14]. In considering machine cognition and the emotional component in CAS, the objective is to implement strategies which enable the processing of contextual information representing external stimuli (problems, threats, or opportunities etc.). For humans the response is predicated on the degree to which individuals generate emotive response to stimuli which may include: ‘happiness’, ‘sadness’, ‘anger’, and ‘fear’ etc [14].

The emotional component is central to understanding emotional response and behavioural choices. Peters [15] identifies roles that emotions can play in behavioural choices including: (1) a guide to information, (2) a selective “attentional” spotlight, (3) a motivator of behaviour, and (4) “a common currency for comparing alternatives”. In the ‘real-world’ there may be multiple roles in play simultaneously. The inverse is also true with cognitive processing generating emotional responses [15], a typical example may be where cognitive dissonance generates discomfort. Moreover discomfort [derived from cognitive dissonance] has been shown to occur not only at the “feeling level” but also multi-modally at the physiological level e.g., in the form of skin conductance responses measured using galvanic skin response (GSR).

Based on this brief overview it is clear that, to include the emotional component, a comprehensive context definition [in intelligent context-aware systems and agents] must employ a multi-modal approach using a broad and diverse range of available CI which must include cognitive data. The objective of AC is the design of computational devices which display innate emotional capabilities and the capability to simulate emotions and empathy, thus context-awareness with AC provides a basis upon which we may effectively enable emotional component in an entities context definition.

3. Context-Awareness and Affective Computing

There is a close relationship between context-awareness and AC in that the emotional component forms an important element in an individual’s current prevailing state (or ‘context’). While the origins of research into AC may be traced as back to the early philosophical studies into emotion [16], the genesis of the modern approach to emotion and AC [as it relates to computer science] may be traced back to the seminal research by Rosalind Picard [17]. AC is a multidisciplinary and interdisciplinary field of research which targets the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. AC is a multi-modal (from a data processing perspective), multidisciplinary, and interdisciplinary field of research which encompasses computer science, psychology, and cognitive science [18].

AC is the relationship to machines where the objective is the design of computational devices which display: (1) “innate emotional capabilities”, or (2) “the capability to simulate emotions and empathy” [18]. A primary motivation for AC is the capability to simulate empathy in machines (including intelligent agents) capable of interpreting humans’ emotional state(s) and adapt the machines’ response to them by providing an appropriate response. Marvin Minsky in [19] has argued that emotions may be related to broader challenges that apply to machine intelligence and has observed that: “in the emotion machine, emotion is not especially different from the processes that we call thinking”. Typically AC has addressed a broad range of domains including: (1) Conversational agents, (2) facial recognition (i.e., happy, sad, confused, etc) systems, (3) e-learning applications, and importantly healthcare systems as discussed in this paper. For a comprehensive overview of AC research and usage see [17,18,20,21].

Context-awareness describes a concept in which the profile of an individual is defined by its context [2], context-awareness employing [a] context to identify an entities current prevailing state. Context is highly domain specific requiring the identification of domain specific function(s) and properties that identify an entities dynamically changing context, therefore in practice almost any information available at the time of an interaction can be viewed as contextual information for use in AC in CAS including: (i) physical conditions, (ii) spatio-temporal information, and (iii) the social situation. For a detailed discussion on context and CI see [2,22]. Such information has formed the basis for the majority of the current context-aware applications and systems. However, research is addressing a much broader and more diverse range of available contextual information. Viewed from an intelligent systems perspective the range of data considered includes:

• Sensory information including sound (hearing and speech), vision (eyesight), and touch including the positions of limbs and fingers etc. in robotics
• Physiological measurements (e.g., blood pressure, heart function, respiration, galvanic skin response, motor functions and facial affects (including muscle activity), and cognitive functions (brain activity)
• Abstract contextual information such as an individuals emotional responses, intuition, feelings, and sensibilities expressed in terms of linguistics and semantic terminologies

Such information is central to the realisation of context-awareness and the implementation of emotion with AC.

Context-aware systems have been developed in recent years to improve user human computer interaction (HCI) in applications and services which operate in highly dynamic environments [23,24]. Contextual information [data processed into information useful in the processing of contextual information] is fundamental in the development of CAS where improved interactions between computers, devices, and entities form a systemic requirement. The range of contextual information identified along with the dynamic nature of the data demonstrates the inherent complexity of context, its domain specific nature, and the difficulty in defining and measuring it. The application of the demands made by AC and the incorporation of emotive response to stimuli only serves to increase the complexity and dynamic nature of CAS.

The genesis of context and CAS dates from the early 1990’s [25]. CAS are designed to target service provision to individuals and entities [2] however the capability to effectively utilize the potential range of CI has been historically limited [2,26] with the domains typically addressed being generally limited to simple recommender systems providing for example tourist information [2,7]. Thus, the development of CAS may be considered to be at an embryonic stage and while an increasing range and scope of data and information is being used in context-aware systems (e.g., see [27,28], emotion (more accurately stated a emotive response) has not been effectively addressed [26] and AC in CAS presents opportunities to implement emotion in intelligent CAS.

4. Research into the Human Brain

On a conceptual modelling level, research into the human brain dates back over may decades and has provided many insights into the functioning of the human brain. Conceptual models of the brain at the neuron level have been proposed (see Figure 1) however, given the developing knowledge of the brain such models demonstrate that knowledge of the brain and its function remains relatively rudimentary and is the result of much (albeit carefully researched and considered) conjecture. However, the research being conducted by the Allen Institute, the Human Brain Project, and the Human Connectome Project show that the conceptual models such as Figure 1 are an accurate (if simplistic) representation of the functioning of a neuron in the brain. Research into the human brain may be considered under two principal headings: (i) large scale research, and (ii) small scale research.

Large scale research is investigating the interconnectivity of the brain on a neuron level. Such projects include principally the Allen Institute for Brain Science [30] which has investigated the working of the human brain at a neuron level, the Human Brain Project [31] which carries out similar complementary research, and the Human Connectome Project [32] which aims to construct a “network map to shed light on the anatomical and functional connectivity within the healthy human brain”. These research projects have produced significant results however it may be observed that the large scale research projects are limited by the capability of current computing systems to map the human brain in it’s ‘static state’ with ‘real-time’ mapping being a distant prospect [33]. Thus, the focus of brain research has been on the mouse brain [30,31] and the hippocampus of the macaque monkey [32]. This research has produced impressive results however translating this research into practical computational cognitive models usable in AC and CAS is challenging and remains an open research question. The large scale research projects are generally open source and provide a large repository of published research and results along with research tools readily available to research groups. There is a prospect of the small scale research projects leveraging these resources to create cognitive models usable in AC and CAS.

Small scale research studies generally investigate the cognitive function in the human brain, for example see [27,34,35,36]. Important work is being carried out using electroencephalography (EEG) [28] and Functional magnetic resonance imaging (fMRI) scanning (see: [27,37]. The small scale research studies have reported significant results in the medical domain, for example see [27,34,38,39,40].

4.1. Cognitive Science and Cognition

Cognitive science and cognitive modelling can be traced back over many decades. Dating back to 1943, Kenneth Craik [41] considered models and noted that the “idea that an organism may make use of an internal model of the world is not new” and if a “small-scale model of external reality and of its possible actions is carried in an organism’s head” alternative courses of action may be evaluated and the optimal reactions to present and future situations may be identified before they arise by utilizing knowledge (or experience) of past events.

Usage of the term ‘cognition’ varies across disciplines however, for the purpose of this paper, it relates to an information processing view of an individual’s cognitive functions. In cognitive psychology and cognitive engineering, cognition is typically assumed to be information processing in a “participant’s or operator’s” mind or brain [42]. Originally grounded in psychology, cognition may be viewed in terms of the set of mental abilities and processes which use existing knowledge and generate new knowledge based on learned experience generated in a feedback-loop over time as modelled in Figure 2 which shows the conceptual view of the continuous information processing and updating. There is an interesting synergy between the ’internal model’ as espoused by Craik in [41] and an individual’s cognitive processes developed over time and the related reactions to both recognised and new phenomena (or stimuli).

Human physiology has a certain components that define an individual, Norman [42] has proposed a conceptual model which shows the components and related interactions. This well accepted view of cognitive functions has resulted in their modelling as separable information processing subsystems (see Figure 3, for a detailed exposition on the components, processes, and interactions see [42]). Additionally, there are sub-systems to implement: (i) internal reasoning and deduction (which includes thought processes), (ii) problem solving, and (iii) language [42]. The model presented by Norman may be viewed in terms of a modern updating of the conventional flow chart of the human information processing system. Figure 3 summarizes the “Pure Cognitive System” which is built around “pure cognitive functioning, with a physical symbol system as its central component”. The basic components are processing systems that use environmental (sensory) information to perform general central processing operations, and control reactions and motor functions. As shown in Figure 3, cognitive processing is complex with a variety of interacting sensory information controlled by a processing structure which, while not fully understood, is known to be capable of simultaneous operation in the areas of: ‘self awareness’, and ‘consciousness’ for some processes. The central functions shown in Figure 3 are “sufficiently vague to allow for a large number of interpretations of it’s nature” [42]. The model proposed by Norman has, based on research currently being undertaken on the neuron level, been shown to be conceptually an accurate model, however it is not an accurate representation of the complexity that exists in the interconnected nature of the connectome [in the brain] and its functioning as the interconnections are not localised in one locus in the brain.

In discussing the philosophy of the mind and cognitive science Shaun Gallagher [43] introduces the concept of ‘self’. There are two concepts of self: the minimal self (“a self devoid of temporal extension”) and the narrative self (which “involves personal identity and continuity across time”). The twin concepts of self illustrate how the philosophical approach can inform cognitive science and suggests that a two-way collaboration [between neuroscience and computer science] may lead to a more fully developed account of self and awareness in entities (e.g., a humanoid robot as discussed in Section 4.2) and computer systems with potential applications in machine cognition. The concept of self may be viewed in terms of an individual’s ‘internalized’ view of the world developed over time based on interactive experience of their ‘externalized’ world. Going further, we must consider the stimuli that prompts an emotional response. There are two types of stimuli: (i) ‘external’ stimuli that affects individuals (this relates to reactive situations which confront an individual in their interaction with their environment), and (ii) ‘internal stimuli (this relates to internally generated actions the individual initiates). In practice, individuals interact with their ’world’ and learned experience gained from their external environment ’feeds’ into the ‘internalized’ self which in turn influences the way humans interact with their world. This process can be viewed as a continuous cognitive information processing feedback loop, Figure 2 shows a high level conceptual view of the process and thus a cognitive model of an individual which may be implemented (at a very basic level) in a computer system which incorporates machine cognition.

4.2. Machine Cognition

In considering machine cognition Mondal [44] poses the question: “Does Computation Reveal Machine Cognition?”. Mondal argues that the nature of machine cognition has been shrouded in “incomprehensibility” and that “human cognition is still faintly understood”. He goes further in arguing that machine cognition is far less understood than human cognition despite the current knowledge relating to computer architectures and computational operations. Human interpretation [of computation] is required whereupon it becomes a type of “semiotic causation” (SC) which gives meaning to computation [44]. SC may be viewed in terms of cognitive conceptual models (CCM) which are cognitive conceptualisations of processes or abstract concepts, for example, conceptualising colour [9]. A CCM arguably has synergy with the concept of semiotics. Generally applied to the media (film and text), semiotics is the study of signs and sign processes in which currently experienced phenomena are interpreted as referring to other, experientially absent, phenomena, thereby becoming meaningful entities, or signs. The reference of a sign is made possible by memories of past interactions with the components of the environment [45].

An interesting perspective on [machine] cognition and cognitive science has been proposed by Simon in [46] where he argues that we may view intelligent systems over three distinct time scales: i.e., ‘short’, ‘medium’, and ‘long’. In the shortest time scale, intelligent systems dynamically change their behaviour in ’real-time’ in response to problem situations whereas for a longer (medium) time scale, such systems apply adaptations that are preserved and remain available to process new phenomena. In effect, such systems learn based on experience as modelled in Figure 2. For the longest time scale, intelligent systems evolve, possibly analogous to nature inspired evolution, based on transmitted inheritance whether biological or social [probably both] which may result in a progressive change in systemic behaviours.

These time scales are interesting from a computerised systems perspective as they relate ‘real-time’ data processing and long term persistent storage of data (which it has been argued may be in fact pre-existing ‘brain states’ [47]). Intelligent agents and CAS function (generally) in ‘real-time’ however data collected over time may be processed in ‘Big-Data’ solutions [48], thus, it may be observed that the scales are not ‘independent’ but are in fact ‘interdependent’. While the short time scale will be used in a dynamic CAS the medium and longer term time scales will over time, based on observation and experience as modelled in Figure 2, inform dynamic behaviour and emotional response to dynamic stimuli in humans. In terms of CAS machine cognition encapsulates mental functions and processes with states of intelligent entities which include: humans, collaborative groups, human organizations, highly autonomous machines, and artificial intelligence.

AC has gained significant traction in the design of computational entities which exhibit either innate emotional capabilities or the capability to convincingly simulating emotions. Such computational entities clearly include ‘intelligent agents’ and ‘humanoid robotics’ (which includes both physical and computational entities) which may be viewed as assistants for human activities which are capable of social interaction. Thus, humanoid robots may be viewed as entities to which AC in CAS apply. Going further, such entities will clearly embody the concept of self and will embody self awareness, an awareness of their external environment, and potentially exhibit empathetic reactions (see Figure 4 for an example of an humanoid robot which fills the role of an empathetic companion and personal assistant).

Interesting studies into cognition and robotics have investigated common metrics for human-robot interactions including the use of cognitive maps for mobile robots. Humanoid robotic systems must incorporate the capability to accommodate dynamically changing states and environments and learn based on the ’feedback-loop’ as modelled in Figure 2. Where humanoid robotics are used to provide interactivity with humans, humans and humanoid robots would ideally share the same cognitive conceptual models of the cognitive functions.

5. Context Modelling

A context definition is essentially a model of an entities current prevailing state, thus context modelling forms a vital role in CAS. Context models must address three key characteristics of every systems model: ‘complexity’, ‘credibility’, and ‘uncertainty’ (which is a function of the fuzzy nature of context). Uncertainty is not fully understood [53], however, it is known that uncertainty plays a pivotal role in efforts to maximize the usefulness of context-aware systems. Additionally, humans express emotions in terms of semantics and linguistics [26,54] which can be effectively implemented using hedge algebras with kansei engineering [54].

Recall that in considering cognitive models Craik [41] noted that a small-scale model of external reality and of its possible actions is carried in an organism’s head. Further, Picard [17,20] observed that AC targets the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. Humans view the world through a unique perceptual filter developed over time based on observation and learned experience [26,55], there is a clear synergy between a humans perceptual view of the world and the observations of Craik and Pickard.

Traditional context-models have reflected the limited scope of context-aware systems and must be extended to include the emotional component. Therefore, data structures and context models must be capable of encapsulating: (i) the traditional concept of a context model, (ii) the additional emotional component, (iii) the demands of AC which is a multi-modal (from a data processing perspective), multidisciplinary, and interdisciplinary field of research which encompasses computer science, psychology, and cognitive science [18], and (iv) the capability to realise semantic and linguistic terminologies. A comprehensive discussion on data structures and context modelling is beyond the scope of this paper, for a detailed discussion see [48,56,57].

In summary, traditional relational database management systems (RDMS) which, whilst historically successful, have significant limitations given the traction being generated in respect of unstructured data (or ‘NoSQL’) [48]. The current optimal approach to the creation of such a data structure is ontology-based context modelling (OBCM) [2,56] as it provides a cross-platform approach which can function as a simple data structure with the capability to implement inference and reasoning. Additionally, using entailment and subsumption ontologies may be merged to create domain specific data structures, e.g., the combining of a person ontology with a domain specific (e.g., medical or pedagogic) ontology.

Research addressing context-awareness and OBCM in a medical context has demonstrated the utility of the approach in respect of physiological signals knowledge representation and as an emotion reasoning model for mental health monitoring. Moreover, Interesting research has investigated NoSQL systems with a ontologies [58] where an ontology is ‘layered’ onto a ‘NoSQL’ approach to provide a semantic capability; this research has reported interesting and potentially useful methods of data modelling in many domains of interest including importantly the medical domain where semantic representation forms an important feature.

6. Implementation Strategy

Thus far, cognitive science and modelling has been considered, this section presents an overview of the proposed implementation strategy. The proposed method [to process contextual information with decision-support] uses ‘context matching’ (CM) which provides an effective basis upon which ‘Partial Matching’ (PM) [based on fuzzy set theory with degrees of set membership] and CS is achieved [2]. Figure 5 shows a high level conceptual view of the CM with a threshold membership function. An alternative term [for membership function] is distribution function, berkan1997fuzzy. The defined thresholds provide a measure of the degree of ‘qualifiedness’ [2] when considered in terms of everyday natural language, this complies with the requirement to address semantics and linguistics as discussed in Section 5 and dscussed in detail in [54].

The proposed approach employs fuzzy set theory which enables PM (degrees of membership of a set). For a comprehensive exposition on fuzzy sets, fuzzy logic, defuzzification, and membership functions see [53]. Fuzzy rule-based systems with decision support are discussed in detail in [59]. Context is domain specific. For example, in the medical domain the intention [of context processing] is to model a presenting patient’s current prevailing state (or context). These function may be summarized as follows:

• Patient diagnosis: to compare a patient’s presenting symptoms (the $input$ context) against the symptoms that classify specific diseases (the $output$ context) in the diagnostic process. This demands that multiple symptoms and test results are evaluated and a confidence-based diagnosis recommendation made in a decision-support system. This may be viewed in terms of iterative context processing as modelled in Figure 6.
• Patient monitoring: to compare changes in a patient’s context at time $\left(t_0\right)$ and $\left(t_1\right)$ (the period between $\left(t_0\right)$ and $\left(t_1\right)$ will be determined by a clinician). The evaluation is designed to identify relative change ($positive$, $negative$, or where there is $no$ change) in a patient’s context and provide the information in a decision-support system to enable the implementation of appropriate intervention(s) as required.

Context-matching is in effect a deterministic search where $\left(u\right)$ is a member of a set $\left(U\right)$, this process is essentially a linear ‘brute-force’ search of the hypothesis space $\left(U\right)$ with the objective of achieving CM. In thinking about medical diagnostics, we aim to match patients observed symptoms and test results to specific conditions. In this case the information will be the tacit and explicit knowledge derived from clinicians in collaborative systems development.

Fuzzy sets are a generalization of ‘crisp’ sets. In crisp sets the ‘characteristic function’ assigns a value of $\left\{0,1\right\}$ to identify ‘members’ and ‘non-members’ respectively. Fuzzy sets provide the functionality to provide gradual transitions from membership to non-membership (and vice versa). This capability has a broad utility as, while it provides a basis for graduated membership transition, it also enables the representation and measurement of uncertainties/vague concepts expressed in natural language. When $\left(A\right)$ is a fuzzy set and $\left(x\right)$ is a relevant object, the proposition: $\left(x\right)$ is a member of $\left(A\right)$ may not be either true or false as is required by two-valued logic. It may be the case that $\left(x\right)$ is a member of $\left(A\right)$ is true only to some degree where the degree to which $\left(x\right)$ is actually a member of $\left(A\right)$.

It is most common (but not required) to express degrees of membership in fuzzy sets (also as degrees of truth of the associated propositions) quantitatively in the closed unit interval $\left[0,1\right]$, The extreme values in this interval, (i.e., $\left[0\right]$ and $\left[1\right]$)1, represent respectively the total ‘denia’l (no membership) and ‘affirmation’ (full membership) of the membership in a given fuzzy set as well as the ‘falsity’ and ‘truth’ of the associated proposition. A fuzzy set can be defined mathematically by assigning to each possible individual in the universe of discourse a value representing its grade of membership in the fuzzy set. This grade corresponds to the degree to which that individual is similar or compatible with the concept represented by the fuzzy set. Thus, individuals may belong in the fuzzy set to a greater or lesser degree as indicated by a larger or smaller membership grade.

Membership of set is often represented by ‘real-number’ (ℜ) values ranging in the closed interval between $\left[0,1\right]$. Membership grades (degrees of full membership and full non-membership in the fuzzy set) can still be indicated by the values of $\left[0\right]$ and $\left[1\right]$ respectively. We may consider the concept of a crisp set to be a restricted case of the more general concept of a fuzzy set for which only these two grades of membership are allowed.

6.1. Alternative Methods

The use of context is increasing across domains and systems and, as observed in this paper, CAS are inherently highly domain specific. In user interactions with a CAS, input data and CI is highly dynamic and unknown prior to the user interacting with a system. Moreover, the output CI (relating to potential context matches as modelled in Figure 6) relates to of a multiple number (perhaps many thousand or even millions) of individuals. As such, there of generally limited or no ‘a-priori’ knowledge regarding the input and output contexts. Additionally, CAS are characterised by fuzzy dynamic environments where sparse data may be incomplete, uncertain, imprecise, vague, and non-specific. The research has considered these characteristics and, along with the proposed approach has considered alternative methods which principally include: (i) Bayesian methods, (ii) inference and reasoning, (iii) decision trees, and (iv) neural systems. Space restricts a detailed analysis and exposition on the alternative methods considered, for a detailed analysis addressing the alternative methods considered see for example [61]. A brief overview of the alternative methods considered is set out below.

However, prior to considering alternative method, it will be useful to briefly outline the nature of the problem being addressed by the proposed implementation strategy as introduced in Section 6. The primary function of a CAS implemented using CP with CM is to provide $predictable$ decision-support; thus, the decision-support problem is $not$ a classical classification problem requiring machine learning techniques typical in artificial intelligence (AI) systems.

The implementation strategy proposed is a fuzzy rule-based approach, the literature demonstrating the efficacy of rule-based systems and fuzzy rule-based systems in a broad range of applications in diverse domains, systems and technologies [53,59]. CP with CM results in a quantitative result in the range $\left[0,1\right]$ as shown in [2] providing an efficient and effective basis upon which $predictable$ decision-support may be realised. Additionally, this result may be viewed as a level of confidence relating to suggested diagnoses in, for example a medical diagnostics decision-support system or in health monitoring scenarios.

6.1.1. Bayes Theorem

Bayesian reasoning provides a probabilistic approach to inference to enable inferred conclusions to be drawn (i.e., the Boolean truth) as to a user(s) suitability for personalised service provision. A practical issue for Bayesian methods is that they require a priori knowledge of prior probabilities [61]. In the context-matching problem the input and output data are (generally) $a-priori$ unknown at the time of a user’s interaction with a CAS as such information is highly dynamic and not generally available. This characteristic [of CAS] calls into question the potential for Bayesian methods to address the demands of CP with CM. An additional issue for Bayesian methods is the potential for significant computational overhead with the resulting tractability issues.

6.1.2. Inference and Reasoning

The proposed implementation strategy as currently constituted does not use inference and reasoning [2]. However, in using OBCM with OWL DL (based on description logic) and has been designed to incorporate the ability to implement inference and reasoning in CP where the domain specific design requirements call for such an approach.

6.1.3. Decision Trees

The posited approach has been designed to enable CM between an input context and (multiple output contexts as modelled in Figure 6. The output of a CAS is a Boolean decision as to the suitability of an entity for personalised service provision. Therefore, implementation essentially reduces to a decision problem. The modelling school of decision analysis often attempts to construct an explicit model of the relationships between the input and potential solutions, often in the form of decision trees (DT) [61].

However, the problems best suited to decision tree solutions have the following characteristics [61]: (i) instances are represented by attribute-value pairs, (ii) the target function has discrete output values, (iii) disjunctive descriptions may be required, (iv) the training data may contain errors, and (v) the training data may contain missing attribute values. However, CP and CM require a broader range of logic functions including conjunctions $\left(AND\right)$. An analysis of the DT approach to decision-support has concluded that inductive inference using DT learning is an inappropriate solution to enable CM with predicable decision-support.

6.1.4. Neural Systems

There is a very large body of research addressing neural systems which includes research-related neural systems and context, for example see [62]. Artificial neural networks (ANN) provide a robust approach for approximating real-valued, discrete-valued and vector-valued target functions.

There are perceives similarities between ANN and the proposed implementation strategy. For example, a perceptron (see Figure 7) shares with the proposed approach the ability to represent Boolean functions and the application of a prioritiztion (weighting) to induce prioritizing bias. Additionally, perceptrons can represent all of the primitive logic functions $\left(AND\right)$, $\left(OR\right)$, $\left(NAND\right)$, and $\left(NOR\right)$. However, in an analysis of neural systems, there are issues relating to the nature of the data processed in user interaction with the system and the computational overhead incurred with respect to ‘real-time’ systems.

6.1.5. Conclusions

Based on the foregoing discussion, the proposed implementation strategy represents the optimal approach in addressing the class of problem this research aims to address, particularly in ‘real-world’ applications and systems where minimizing the computational overhead in ‘real-time’ systems’ is an important factor in the selection of appropriate methods.

Where classification forms a central function and computational overhead is less relevant, alternative approaches may prove to be interesting either in single or hybrid methodologies. For example, in research into mental health issues based on EEG sensor data, we have used a range of methods including decision trees in combination with the naive Bayes classifier. However, as observed, the decision-support problem is not a classical classification problem and the posited implementation strategy has been identified as the optimal approach.

7. Discussion

The aim of intelligent systems is to realise set goals; this may be reflected in meeting defined goals for entities in dynamic environments. Clearly, context and context-awareness, while interesting from a research perspective, must utilise technologies applicable in the ‘real-world’ domains such as healthcare and pedagogic systems. CAS must be adaptive to dynamic environments and are inherently domain specific [2]. Systems that are adaptive may equally well be described as “artificial” [46] for, as environments change, systems must change to match the dynamic ’states’ (or context) and thus ’mirror’ the new situation [2,65]. In the healthcare domain context-awareness must enable adaptation to dynamic and uncertain environments [22].

Context-awareness is characterised by its fuzzy nature and uncertainty [22], these characteristics are reflected in the ’real-world’ and context modelling must accommodate these characteristics while including the emotional component. This is reflected in the approach introduced in this paper which is, in summary, an event driven fuzzy rule-based system implemented with OBCM as discussed in [2]. This approach can be viewed in terms of a system designed to provide an effective basis upon which personalized services [generalisable across a broad and diverse range of domains, systems, and technologies] may be achieved under uncertainty while accommodating (or at least mitigating violations) of CS . For a detailed discussion on CP, CM, and CS with an analysis and an example implementation [based on pedagogic systems in the domain of tertiary education] see [2,11], a discussion on data structures including structured and unstructured data and ’NoSQL’ database technologies may be found in [48].

Data (or CI) comprise a systemic requirement for CAS, and the desire for AC to recognize, interpret, process and simulate human affect requires a broad and diverse range of data, including cognitive features. Accommodating the range of potential CI demands a multi-modal and multi-factorial approach capable of managing the uncertainty and fuzzy nature of context, which is a characteristic of the ‘real-world’ in general and clearly applies to the medical domain [22]. The proposed approach when used with OBCM has been shown to be effective in the domain of tertiary education [2,66] and, as observed, will generalize to the healthcare domain. However, while the technical issues related to the processing of the environmental and physiological data have generally been addressed, the implementation of emotion and emotive response (in CAS with AC technologies) remains a challenge largely due to the need for effective non-invasive sensor-based data capture (of the cognitive features related to the emotional component).

In considering cognitive science and machine cognition [in intelligent systems and agents] the relationship that exists between intelligent CAS, AC, and the emotional component has been identified. It is argued that there exists a synergy between intelligent CAS and AC in that, when viewed from a computational perspective, the objective of AC is to enable machine cognition and realise emotional capabilities and/or simulate emotions in entities such as intelligent agents which include intelligent empathetic assistants (see Figure 4) and robotic systems (which may be humanoid or software entities). The creation of CCM capable of modelling cognitive processes and emotion forms a central role in the development of intelligent agents. However, while research into the development of such models may be traced back over many decades (see for example [41] and Section 4.1), on a computational modelling level much of the published research may be considered to be informed conjecture.

Research into the human brain may viewed on two levels: (i) research on the neuron and connectome level [30,31,32], and (ii) research into mental disorders and depression using Electroencephalography (EEG) [27] with Functional magnetic resonance imaging (fMRI) [28,67]. Research on the neuron and connectome level is generally conducted in large-scale projects while research into mental disorders (such as depression) is generally on a smaller scale. The current state-of-the-art as in terms of a comprehensive understanding [of the human brain] is currently limited as it relates the development of comprehensive conceptual models [of the human brain] for both research purposes and for use in translational research [68] represents a significant challenge and remains an open research question. The results derived from the on-going research into the human brain both at a neuron level and the results obtained from the on-going investigations using EEG and fMRI will inform the development of comprehensive conceptual models of the human brain which may provide a roadmap for the integration of AC and the emotional component in intelligent CAS.

In considering future directions for research into intelligent CAS systems capable of managing the multi-modal nature of the contextual information [related to both the generality of context and the additional data that relates to the AC and the component] it may be noted that:

• The approach to enable the effective CP is known and understood [2,50].
• The issues related to data structures for CAS are understood and the current optimal approach to context modelling of structured data to enable ‘persistent’ and ‘in memory’ data storage has been identified as OBCM [56,67,69,70].
• The management of unstructured data is known and understood [48]. Moreover, research has investigated data integration between ‘NoSQL’ data stores and RDMS into a single virtualized database [58], this holds the potential to address the limitations inherent in both approaches.
• There are interesting developments in object oriented database systems implemented with relational database management systems [71] which form and interesting direction for research. Such developments are reflecrd in support for ontology-based semantic matching in RDBMS, an area that offers interesting potential opportunities to use OBCM in concert with RDMS [72].

The challenges lie less in the approach to data structures and implementation strategies but to the availability of effective cognitive conceptual models of the human brain [36,37]. It may also be observed that research into cognitive processing has made significant progress in for example the diagnosis and management of depression [27,34,35,38,39,67,73,74,75].

However much of the data used in assessment of mental disorders is generally very ‘subjective’ and is frequently based on self reporting when considering depressive states which may be viewed in terms of a spectrum as discussed in [22,76]. In considering schizophrenia [40] diagnoses similarly utilise subjective data and for Alzheimer’s disease and dementias the monitoring of the development of the condition frequently uses scales such as the well known and recognised Cohen Mansfield Agitation Index to asses the Behavioural and Psychological Symptoms of Dementia [77,78,79]. Current research, for example see [27,34,35,36,37,38,39,67,73,74,75], is attempting to address the provision of objective data usable in the diagnosis and management of mental disorders in ‘real-world’ conditions.

There are interesting developments in the field of Transcranial Magnetic Stimulation (TMS), the research providing promising results in studies of depression and insomnia; however this field of research remains at an embryonic stage. There is a clear demand for the development of effective non-invasive sensor technologies to capture the multi-factorial objective data which may be processed in CAS, this remains an open research question.

It may be concluded that addressing the challenges identified requires a multi-disciplinary approach where computer scientists (who generally lack medical, psychological, and physiological expertise) must collaborate with medical experts (who generally lack the required computer science knowledge) to create the required technological solutions and data structures which incorporate AC and the emotional component.

8. Concluding Observations

It has been argued in this paper that context-aware systems, as currently conceived, fail to effectively address the integration of AC and emotional response and have limited computational intelligence. In response to the question posed in this paper (namely, do we understand the relationship between affective computing, emotion and context-aware systems?), the answer is yes. However, it must be recognized that there remain significant challenges in terms of effectively modelling the human brain and in measuring cognitive states.

The challenges are recognized as significant, difficult and non-trivial. However, notwithstanding the scale of the challenges, it is posited that the integration of AC and the emotional component in CAS provides the potential to increase the complexity of cognitive architectures for computer systems and improve the level of computational intelligence in intelligent CAS. Notwithstanding the challenges, effective implementation of emotion and emotional response for entities in intelligent CAS systems provides exciting potential opportunities in a broad range of domains including importantly healthcare and pedagogic systems to the benefit of all stakeholders.