Human Computation to Enhance E-Service Consumption among Elderlies ^†

Abstract

Smart Cities aim to increase citizens’ quality of life. Smart Government is a part of Smart Cities domain aiming to enhance the communication and interactions of citizens and companies with government. The SIMPATICO framework combines machine and human intelligence to simplify e-services i.e., web accessible apps to accomplish administrative procedures online, and thus enable a more widespread adoption of electronic procedures. This paper reviews the impact of instrumenting e-services, in Galicia region in Spain, with SIMPATICO features, e.g., text and workflow simplication, autocompletion of personal data and public procedure documentation and support through crowdsourced questions and answers. Particularly, taking into account the lessons learnt at the first pilot evaluation with Elderly people, this paper describes the public procedure model proposed by SIMPATICO which backs the application of diverse strategies to better support users while facing the completion of complex administrative procedures.

1. Introduction

In these last years, the squeeze for public funding has increased the need for the Public Administrations (PAs) to improve the efficiency, effectiveness and quality of its online public services (from now on, e-services). However, despite the effort of these past years to modernize processes, policies and create innovative services for the citizens, the adoption of e-services in the EU28 has been flat. Among the citizens needing to submit forms to the public administrations in 2016, 13% used the offline channel because they did not use Internet at all. Of the remaining 87%, all of them Internet users, 52% chose the online channel, while 35% preferred an offline interaction with the public authorities. This marks considerable progress compared with 2011 when only 39% of the population was sending completed forms online, and the percentage of those who couldn’t because they were not Internet users was of 21% [1].

In addition to complexity, one of the main problems of current e-services is that they are built following an administration-centric approach rather than according to the citizens’ needs (user-centric approach). This problem directly translates into a low uptake. For example, despite the large and increasing availability of e-services, users’ poor computer literacy, their low familiarity with administrative procedures, and their limited understanding of what concerns legal and technical jargon, can make comprehending these online services and the accompanying documentation very difficult if not impossible. The same is often true for companies, in particular micro and small enterprises, faced with highly complex procedures and documents outside their field of expertise.

The SIMPATICO project’s (https://www.simpatico-project.eu/) goal is to improve the experience of citizens and companies in their daily interactions with the public administration by providing a personalized delivery of e-services based on advanced cognitive system technologies and by promoting an active engagement of people for the continuous improvement of the interaction with these services. The SIMPATICO approach is realized through a platform that can be deployed on top of an existing Public Administration (PA) system and allows for a personalized service delivery without having to change or replace its internal systems: a process often too expensive for a public administration, especially considering the cuts in resources imposed by the current economic situation.

The goal of SIMPATICO is accomplished through a solution based on the interplay of language processing, machine learning and the wisdom of the crowd (represented by citizens, business organizations and civil servants) to change for the better the way citizens interact with the PA. SIMPATICO adapts the interaction process to the characteristics of each user profile; simplifies text and documents to make them understandable; enables feedback for the users on problems and difficulties in the interaction; engages civil servants, citizens and professionals so as to make use of their knowledge and integrate it in the system (Figure 1).

The structure of the paper is as follows. Firstly, in Section 2 an overview of related projects also attempting to democratize e-services access and thus enhance their take up will be offered. Section 3 will continue with an overview of the SIMPATICO project, its two main goals and the devised underlying architecture. Section 4 will explore the public procedure metamodel proposed by the project, where the different elements that constitute a public procedure are described, followed by its mapping to the Citizenpedia component, a tool to document and support with human contributed knoweldge public procedures. Section 5 will cover how the SIMPATICO solution has been evaluated through two instrumented e-services trialled by elderly and disabled people in the region of Galicia in Spain, during pilot phase I. In preparation of phase II, this very Section 5 also describes an in-lab assessment about the complementarity of the three components conforming the Citizenpedia. Finally, Section 6 concludes outlining further work and drawing some conclusions.

2. Related Work

The concentration of resources and services around cities offers new opportunities to be exploited. Smart Cities [2] are emerging as a paradigm to take advantage of these opportunities to improve their citizens lives. E-governments are essential in order to have Smart Cities. Electronic prescription smart cards and e-ID [3] are examples of tools created by governments to introduce e-governments to the citizenship.

Smart Governments are also required in Smart Cities in order to provide useful information to citizens. Open Data provision and support helps governments in the following three aspects [4]: Transparency, Releasing social and commercial value and Participatory Governance. Reference [4] also presents Open Government data life-cycle and discusses about Data Formats, Data Ambiguity, Data Discoverability, Overlapping Scope, Data Representation and Public Participation.

Public Participation gives a chance to the stakeholders to suggest what to publish and to rate datasets or features of a Open Government Data portal. Collaboration goes further than participation and permits the cooperation amongst different stakeholders in order to create new data, for example, through a questions and answer portal. There are different studies centered in Public Participation and Collaboration. [5] analyzes the published data, its relevance and usefulness. In [6] the authors attempt to identify which are the factors that influence the stakeholders to create and consume Open Data. There are initiatives whose goal is to increase all the stakeholders participation [7]. Many tools have been created with the objective of enhancing the Public Participation. For instance, the Query mapper [8] is a good example of it. It is a component of the IES Cities platform devised to democratize the development of Open Data-based mobile urban apps.

The way administrative procedures and e-services are represented is a key issue for e-governments. How this representation is carried out is not as trivial as it look like, since the data used for this representation is obtained from different sources and not with the same format. There are different studies that try to address this interoperability issue [9,10,11]. In fact, the European Commission adopted on 23 March 2017 the European Interoperability Framework (EIF) [12]. This framework gives specific guidance on how to set up interoperable digital services. EIF provides 47 concrete recommendations that will improve the quality of European public services. This will ensure that services are accessible across countries and policy areas. In other words, they will apply interoperability in practice. This way, public administrations can save time, reduce costs, increase transparency, and improve the quality of services that they offer to citizens and businesses.

3. The SIMPATICO Project

In this paper, we describe our effort to model and make more understandable public services so that their take up can be extended to all publics. SIMPATICO aims to enhance the take-up of existing e-services by adopting a two-fold approach:

• Adapt the interaction process with respect to the profile of each citizen and company (PA service consumer) in order to make it clear, understandable and easy to follow. For that, it proposes the following technical contributions:

  -

  A text adaptation framework based on a rich text information layer and machine learning algorithms [13,14] capable of inducing general text adaptation operations from few examples, and customizing these adaptations to the user profiles.

  -

  A workflow adaptation engine that takes user characteristics and tailors the interaction according to the user’s profile and needs.

  -

  A feedback and annotation mechanism that gives users the possibility to visualize, rate, comment, annotate and document the interaction process (e.g., underlying the most difficult steps), so as to provide valuable feedback to the PA, further refine the adaptation process and enrich the interaction.

• Exploit the wisdom of the crowd to enhance the entire e-service interaction process. For that, it contributes with the following components:

  -

  An advanced web-based social question answering engine where citizens, companies and civil servants discuss and suggest potential solutions and interpretations for the most problematic procedures and concepts.

  -

  A collective knowledge database on e-services used to simplify these services and improve their understanding.

  -

  A gamification engine that engages users and incentivizes them to collaborate by giving them reputation (a valuable asset for professionals and organizations) and privileges (for the governance of Citizenpedia – a new public domain resource) according to their contributions.

Figure 2 illustrates the three logical layers in which the functionality of the SIMPATICO solution has been divided, namely (a) interaction adaption and personalization; (b) front-end interaction and enrichment; and (c) human computation in green. The diagram shows the data flows produced by SIMPATICO stakeholders (civil servants and citizens mainly) when interacting with the distinct components associated to the mentioned layers.

Interaction adaptation and personalization layer (pink colour components). One of the core objectives of the SIMPATICO project is to provide with new technologies that assist users in comprehending the content of public administration websites and documents. For that purpose, SIMPATICO features the Text and Workflow Adaptation Engine, which are responsible for producing simplified versions of webpages, forms, questionnaires and documents in the public administration domain. These components interact with the user and their profile with respect to their personal information in order to adapt to their interaction adaptation needs. With this information at hand, the Text and Workflow Adaptation Engine can help the user in a personalized fashion that adjusts to their profile specifically. The Citizen Data Vaultis the repository of the citizen (or company) personal data, profile and information. It is continuously updated through each citizen interaction with e-services and is used to automatically fill in forms. In this way, the citizen will give to the PA each information only once, as the information will be stored in the vault and used in all the following interactions.

Front-end interaction and enrichment layer (yellow colour components). A second core objective of SIMPATICO is the improvement of the interaction process for each involved actor in the system (citizens, civil servants, companies). For that, SIMPATICO contributes with components for: (a) presenting the user with a fully featured Interactive Front-End for the SIMPATICO e-services, (b) provide the system’s analysis modules with data gathered from the interaction to propose improvements in future versions of the e-services and finally (c) a method of summarizing these proposed changes and provide the civil servants with useful information to improve the services. The Authoring Support Tool (AST) represents a set of instruments made available to the Civil Servant for creating, managing, and maintaining the e-services, their descriptions, simplifications, and model. Meanwhile, the e-Service Monitor (eSM)gets the available processed data from the interaction analysis pipeline (stored in the Interaction Data Log, processed by the Data Analysis component) and produces summaries of interaction issues that are presented to the Civil Servant so that the current status regarding a given e-service is quickly understood, and future versions of the interaction are optimized.

Human Computation layer (green colour components). The components belonging to this layer enable to leverage from the collaborative knowledge provided by citizens and civil servants when editing and improving or consuming e-services, respectively. This layer defines an environment where citizens can solve their doubts and actively take part in the enhancement of the e-services. It exposes several tools for citizens and civil servants. Through the Question Answering Engine, citizens can post doubts and look up for terms and definitions. From the civil servants’ side, this component is an easy channel to rapidly solve doubts. The Collaborative Procedure Designer describes current e-services in the form of flowcharts/diagrams and citizens are able to comment on them. Using the comments of the citizens posted in Collaborative Procedure Designer and the analytics provided by the eSM component, civil servants can enhance e-services. Moreover, the Gamification Engine is a tool that can engage citizens and civil servants in their own interaction through games. Section 4.2 Citizenpedia: Human-Computation framework in SIMPATICO expands on the description of these components, main focus of this paper.

4. SIMPATICO Public Procedure Model

All the components presented in the previous section share a common knowledge model for representing public procedures and the interactions that stakeholders can have with them. To make easier the representation of all the data and the interoperability among components above mentioned, an interaction model has been created.

4.1. Interaction Model

In the Figure 3 all the concepts needed to represent an administrative procedure are shown. This model also has some concepts that are not part of the PA domain language which are required by SIMPATICO components in order to improve the administrative procedures. The interaction between concepts are also represented in the figure.

An administrative procedure is the most generic concept of the model and is composed of different steps that have to be accomplished in order to complete the procedure. To accomplish a step, one or more e-services could be needed. An e-service is a web page through which a citizen fills one or more forms in order to complete a step. E-services are composed of Forms, Blocks, Fields and Paragraphs (Phrase and Words).

In order to add more information to procedures and e-services, and all the components they are compose of, new concepts have been modeled. The description of these concepts and relations between them are explained in

Table 1. Modeled concepts, descriptions and interactions.

Concept	Description	Relations
Question	Question asked by a citizen about an specific modeled concept.	A question can be done about a E-service, Form, Paragraph, Block, Procedure or Procedure step
Answer	Answer to a question created. A question could have multiple answers	An answer is given to a question that is related to an E-service, Form, Paragraph, Block, Procedure or Procedure step
Term	A keyword used to describe an element	A term is related to a question or an answer
Word Definition	A result of text enrichment applied to a single word/phrase	Related to a word or phrase from a paragraph
Simplified Text	A result of text simplification applied to a phrase or paragraph	Simplification of phrase or paragraph
Interaction Workflow Model	A simplified/adapted model of completion of a form as a result of a workflow adaptation.	It is related to a form with one or more blocks
User Personal Data Element	Information about the user	It is closely related to the form through the Interaction Workflow Model
e-service description	Detailed description of an e-service	As it is composed of paragraphs, it is related to all the concepts that interact with the Paragraph concept

.

This new model in which the static information about the administrative procedures is enriched with additional information provided by citizens or algorithms, can be used in order to enhance with SIMPATICO functionality the e-services and procedures PA offers.

4.2. Citizenpedia: A Human-Computation Framework

Based on the interaction model, the Citizenpedia component has been implemented inside the SIMPATICO project. It aims to complement e-government environments with a collaborative space where citizens and civil servants share and exploit accessible knowledge about public procedures, and more specifically, where citizens can solve their doubts and actively take part in the enhancement of e-services. To that end, Citizenpedia has been designed, a participation (one of the key pillars of Open Government) fostering component, as an aggregation of three complementary tools, which are described next.

4.2.1. Question & Answer Engine

The Question & Answer Engine (QAE) (https://simpatico.hi-iberia.es:4569/qae/) is a tool which provides a mechanism where citizens post and resolve doubts regarding e-services and public administrative procedures. The chosen look-and-feel for this tool is similar to other popular question & answer ones, for example Quora (https://www.quora.com/).

The main functionality of QAE is to create and answer questions following a crowdsourcing approach. Users are encouraged to contribute contents in a public manner, with the goal of keeping the generated information to remain over the time. This is usual in QAE places in the field of engineering (e.g., Stack Overflow (https://stackoverflow.com/)), where sometimes an answer written two or three years past in time is useful for the user looking for a doubt. In addition, questions are searchable and sortable.

The QAE design considers to have two main roles: user and moderator. Initially, every citizen is a user and every civil servant is a moderator. Users can search along Citizenpedia and post content. Moderators have higher privileges, i.e., permission to edit/delete contents from other users, addressing the problem of low quality contents sometimes resulting from user contributions.

In order to keep the user engaged, a rewarding and reputation mechanism is considered. Each time a user conducts an action (e.g., posting/answering a question, leaving a comment, rating an answer, ...), it is recorded and several points are granted. Once certain amount of points are exceeded, badges are given. This enables users to gain reputation and distinguish the most active participants in the community. In addition, we consider that once a user reaches certain level of reputation, he/she gains moderation rights.

4.2.2. Collaborative Procedure Designer

The Collaborative Procedure Designer (CPD) (https://simpatico.hi-iberia.es:4570/cpd/es/procedures) is a tool to describe current administrative procedures in the form of flowcharts/diagrams, that enables citizens to comment on them. The core of the CPD is a model that allows the definition of multiple hierarchical views, each one providing a representation of the procedure with a growing level of detail.

Currently, the CPD offers two views to describe the procedures: the value-chain view and the interaction view. The value-chain provides information concerning the sequential phases that the procedure is broken into. If an e-service needs to be used in a procedure phase, information about it is also given. From this view, it is possible to understand the name of the phases and realize the phases’ temporal order. By expanding a specific phase, an interaction view gets displayed. This view shows a flow of the interactions between the citizen and the PA that are carried out in that phase. For any interaction, it is pointed out who of the two actors is the interaction’s initiator. Also, the communication channel through which the interaction will occur is explicitly indicated by means of specific icons.

4.2.3. Servicepedia

Servicepedia (https://simpatico.morelab.deusto.es/servicepedia/) integrates the descriptions of the e-services with the information on the QAE and CPD to have all the elements contextualized. This tools is the one which uses, directly or indirectly, most of the concepts of the interaction model. Thanks to this implementation, Citizens can access to information about procedures, procedure steps, e-service, e-service description forms, fields, blocks, paragraphs and interaction workflow model. In addition, citizens have the opportunity to see all the questions, answers and terms related to all these elements. It offer a simple to grasp birds eye view of administrative procedures.

4.2.4. Architecture

These three main components interact among them and also make use of the following complementary components:

• Gamification: This module permits define games in order to engage citizens to use the components.
• Database: All the information of the components is stored a shared MongoDB database.
• Indexing: In order to perform more efficient searches, the database is indexed using ElasticSearch. This way, for example, during search process typos errors in the search text are managed.
• REST API: Citizenpedia offers an API to expose the key functionalities implemented in the three main components third party platforms.

In the Figure 4 the Citizenpedia architecture with its main and complementary components is shown.

5. Assessing SIMPATICO in Galicia Pilot

In this section the description and results of the two evaluations during the Galicia Pilot in which Citizenpedia is involved are described.

5.1. Evaluating SIMPATICO in 1st Pilot Phase

During this evaluation phase all the components of the SIMPATICO project were evaluated. Regarding the Citizenpedia the main goal was the validation of the QAE sub-component as useful participatory portal to complement e-services’ available documentation with citizen and civil servants’ contributions. In these sessions, citizens from three associations, FEGAUS (https://fegaus.com/), COGAMI (http://www.cogami.gal/) and ATEGAL (https://www.ategal.com/), took part. There were a total of 204 participants with the following profiles:

• 123 elderly citizens: retired people, at least 65 years old, with low to medium IT skills.
• 65 citizens with some kind of disability: people at least 18 years old, with medium to high IT skills.
• 16 civil servants from Xunta.

During the evaluation we arranged participants in groups of 10 to 30 people and conducted several sessions that followed this structure:

1.

Introduction to the session (~5 min):

2.

Warm-up tasks (~15 min): Participants were asked to fill an initial survey with demographic questions, such as their age, IT skills, etc.

3.

Training on SIMPATICO tools (~25 min): A facilitator of the session presented the different tools developed as part of the SIMPATICO project, including the Citizenpedia. In the first phase, the Servicepedia component was not implemented yet.

4.

Complete the e-service using SIMPATICO tools (~15 min): Participants were asked to fulfill electronic procedure use case, being the aim to submit a request for the offered service. They were encouraged to use the SIMPATICO tools in order to support them in the process

5.

Ending survey (~10 min): Participants were asked to complete a survey with questions about their experience using the tools.

As an overall conclusion from the evaluation of Citizenpedia with users, we can state that there is plenty of room for improvement in order to make it more appealing, acceptable, for the users. Participants stressed the difficulties of navigating (jumping) from Xunta Portal (https://simpatico.hi-iberia.es:4570/IFE/BS607A_es.html) to QAE Portal and vice versa. When users clicked on the “Add new question” button from e-service description, at Xunta Portal, a new window was opened in which the QAE Portal is shown. The fact of having two windows opened in parallel confuses the user. Furthermore, although questions and answers were contextualized in the QAE portal by using tags, if they wanted to know exactly where the question was created (CPD, IFE, etc) the QAE redirects the user to that component. This redirection sometimes was impossible to perform because the user had to be authenticated in order to see the procedure element in which the question was created.

Civil servants have found the tool promising, but we need further development in software features to ensure that it becomes a useful tool for the citizens. Special attention shall be paid to the design of the user interface, for the elderly persons to find it more accesible and amenable, and, thus, more acceptable.

5.2. Evaluating Human Computation Tools

Based on the feedback from the first phase, the creation of Servicepedia tool was proposed in order to solve the information descontextualization issue. Once the Servicepedia was implemented and the rest of Citizenpedia components were improved, a new evaluation has been carried out to test the usefulness and usability of all the Citizenpedia components again. Regarding the usefulness, the main question to be answered was if Citizenpedia’s components and Xunta’s web pages are complementaries or if they simply offer the same information and functionalities. In terms of usability, comprehension and simplicity of accessing to data and functionalities offered by components were evaluated.

5.2.1. Experiment Description

The experiment was carried out in a controlled environment, simulating the context of use of the tools. Users were male and females of 22–35 year-old, and they had enough skills to use web-based tools and methods. The test was voluntary and users were aware about their rights. Involved persons were encouraged to think aloud. Although the video and audio of the session was not recorded, the facilitator registered main insights and ideas that emerged at the session. Three different groups of 3 people participated in the experiment, facilitating data gathering and the analysis due the small size of the groups. Each group used a different component in order to realize the experiment as it is shown in the

Table 2. Experiment groups.

Group	Used Component	Number of Participants
Group A	Xunta’s webpage	3
Group B	CPD	3
Group C	Servicepedia	3

.

The process carried out during the experimentation in the three groups was the same with the only difference of the component used to get information about the administrative procedure and the e-service. The administrative procedure used was “BS607A - Programa Bienestar en Balnearios”. which gives the possibility for citizens to apply for a subsidy to go to spas. The process was divided in the following steps.

• Session explanation: The first step to face the experiment was to explain and inform users about the Simpatico project and the procedure of the experiment. For that, after a short welcoming and introduction, the project and the aim of the session were clearly explained. Besides, the users were informed about their rights and all the questions and issues related to the experiment were answered.
• Read the information about the administrative procedure using the component assigned: To start the experiment, each group reads the information about the administrative procedure using their laptops to open the component assigned to them. All components somehow show information about the phases an administrative procedure is compose of and the e-services and information needed to complete them. The duration assigned of this step must last five or less minutes.
• Answer to questions related to procedure descriptions: In order to evaluate if all the information about the administrative procedure is represented, the way it is represented in the components, and the comprehension of it, five short questions were asked to the participants:
  • Is the public procedure opened?
  • Which is the documentation that must be presented?
  • How many steps are necessary to accomplish the BS607A procedure?
  • Which is the name of the first step of the BS607A procedure?
  • How many of the step of the BS607A procedure need to be accomplished using an e-service?
  The maximum duration assigned to this step was five minutes, and it helps to focus users in the experiment and to familiarize with the tool.
• Consume the e-service: After answering to first questions, each group consumes the e-service using the assigned component. Duration: 5 min
• Questionnaire about usability: In order to evaluate the comprehension and usability of the system, the users had to fill the System Usability Scale (SUS) Questionnaire [15]. This tool is a ten-item-based questionnaire and it is commonly used to analyze the usability, usefulness and comprehension of interfaces. The duration of this phase of experiment was 3 min.
• Open questions about the component and the evaluation process: Finally, to extract qualitative data and insights, some open questions were introduced, to give the participants the opportunity of giving their opinion and insights about the interface and system. The duration of this step was of 5 min.

5.2.2. Results and Conclusions

The results obtained during this in-lab evaluation have been divided and analyzed from procedure compression, usability and qualitative results perspective.

For the results related to the compression of the procedure and e-service, answers to the five questions mentioned in the experiment description have been taken as input. Regarding questions about the documentation needed to complete the e-service and the one that asks if the call was opened, participants of all the groups answered correctly. Deviation came with questions about the procedure phases and e-services. Participants who used the Xunta Portal did not find or did not understand this information. Users who used the CPD or Servicepedia did not had problem to find the number of phases neither to identify the first phase name. The difference came with the question about how many e-services, 1 out of 3 participants who used the CPD answered correctly and 2 out of 3 participants using Servicepedia gave a correct answer. These results show how information in the both Citizenpedia components is easier to understand than the original one in the Xunta’s Portal.

Regarding usability, quantitative results obtained using the SUS calculation procedure [15] are presented in

Table 3. This table shows the results of the SUS questionnaire.

	Xunta Portal			CPD			Servicepedia
	User 1	User 2	User 3	User4	User 5	User 6	User 7	User 8	User 9
SUS Result	67.5	42.5	55	50	72.5	70	85	55	85

and

Table 4. Average SUS result by components.

Xunta Portal	CPD	Servicepedia
55	64.16	75

:

The SUS questionnaire revealed promising information, taking into account comparation among the scores of the different components used during the test. Although the average score of Xuntal Portal was 55, which is over 50 and is satisfactory in a certain way, the Xunta Portal obtained the lowest score (42.5 was the lowest score obtained related to the Xunta Portal) among all the alternative support options. This means that this website can be difficult to understand and navigate, and that should be improved to obtain more agile and efficient procedures and e-services. For that, other components that aim at improving the efficiency of Xunta’s Portal, as CPD and Servicepedia had been implemented and evaluated to find if can improve the comprehension, usefulness and efficiency of the Xunta’s Portal and e-services.

The CPD was the second component tested, obtaining higher average score than Xunta Portal: 64.16. Besides, the scores obtained of two users (of three) who tested this component, were over 70 (see Table 3). This means that the CPD improves the usability of the system, and that it is more easy to use that the Xunta Portal.

The final component tested, Servicepedia, obtained an average score of 75, being the best scored of all the components used in the evaluation phase. Two of the three scores of Servicepedia were above 80 which means that, although the system needs improvements, the component is valuable and understandable for the end-users.

Thus, the results of the SUS questionnaire show that Servicepedia was the most usable and understandable component and that helps users to get better interaction with e-services, enhancing the usability of the processes and involving the users with the system.

The qualitative information extracted from the open questions was related about the usefulness, legibility and comprehension of the different components.

The users who used the Xunta Portal stressed that the information to the procedure was too technical and suggested to add a help button or additional and complementary information to understand the confusing terms.

The CPD was perceived as a promising tool. Some users pointed out that separating the process into phases encouraged the learning of the tool. In addition, the split of the process in clear and concise steps makes it faster and easier to understand, involving less cognitive load. Negative insights emerged were related to the content provided by the component. One user stated that there was excesive information and that the reading process was tedious. Therefore, the need of the improvements on the legibility of the text and its hierarchy must be reviewed, guiding the reading of the content in an easy and intuitive way taking into account the diversity of the end-users.

The most relevant idea emerged on the qualitative evaluation of the last component tested, Servicepedia, was related to the importance of the information structuring. The users revealed that the division of the infomation did the comprehension more fluent and improved the legibility of the content. Besides, the identification of the different stages of the process was appreciated, because the users could understand the different steps of the procedure and was easier to understand the flow of the navigation through the interface. They also appreciated the detailed information about the e-service elements. The negative ideas related to the Servicepedia were about the arrangement of the text in column, because this difficults the reading of the content. For that, the improvement of the design of the text field would be an implementation to face in the next iteration of the development. Another suggestion by the users was the implementation of the information in a diagram to understand the main ideas in an easy and fast way.

Participants also gave their opinion about the form they filled in after retrieving information about the procedure and e-service. They had the feeling that there were to many unnecessary field to fill. They suggested to add some explanation of the fields or an interactive guide that helps you to better understand the form.

To summarize, we can conclude that the two tools created by SIMPATICO project, CPD and Servicepedia, offer more information of the overall administrative procedure. The differences between these tools are in the functionalities they have and the usability of them. Analyzing the answers of the users, Servicepedia has more interesting functionalities that give to the user the possibility of accessing to the information about the administrative procedures and e-services in different ways. Regarding to the usability besides of getting a better SUS score, we also got suggestions of what could be changed of added in order to improve this aspect. To conclude we can say that although Servicepedia is the newest component and a lot of improvements has to be done on it, it is already the most useful tool to contextualize and expose to user information about administrative procedure.

6. Conclusions and Future Work

In this paper we have presented a model to represent administrative procedures and the Citizenpedia component which makes use of it. The early evaluation of two of the Citizenpedia tools has demonstrated that Servicepedia, the tool which uses the SIMPATICO public procedure model in a more thorough manner, is the most useful one. Even so, we realized that most of improvements to be done are related to usability and inclusiveness, since users did not know how to access certain features. Taking this into account, one of our future working lines is to carry out a usability and inclusiveness evaluation in order to detect what to improve.

Based on the users feedback, we have also realized that one of main problem are the contents of public e-services themselves. Most of them are not easily understandable, and therefore are not useful and usable, and should be redesigned by their owners, i.e., the public administrations. Even so, SIMPATICO project offers the Workflow Adaptation Engine component which can be integrated into e-services to guide the user when filling out a form and simplify the workflow carried out in their interaction with the e-service.

A second phase of evaluation is going to be carried out in SIMPATICO project. The idea for the second phase evaluation sessions is to mantain the same methodology used in the first phase. We want to analyze if the improvements done to the SIMPATICO solution, including the creation of the Servicepedia tool, make public procedures more understable and, thus, acceptable comparing with the results obtained in the first phase.