Relevant Aspects for an EF3-Evaluation of E-Cognocracy

Abstract

The search for an appropriate response to the new challenges and needs posed by the Knowledge Society in the area of public decisions has led to the development of a number of participation models whose value must be assessed and analysed in an integral manner. Using a theoretical model based on structural equations, the present work identifies the relevant factors for an EF3-approach to the democracy model named e-Cognocracy: it comprises a conjoint evaluation of its effectiveness (doing what is right), efficacy (achieving goals) and efficiency (doing things correctly). The model was applied to a real-life e-Cognocracy experience undertaken in the municipality of Cadrete, Zaragoza. The evaluation resulted in the extraction and identification of a series of relationships that allow the advancement of an EF3-participation acceptance model, in line with the TAM model of Davis and the work of Delone and MacLean, which can be used for the integral evaluation of any e-participation model.

1. Introduction

As exemplified by the social protest movements of 2010 (e.g., the revolution in Arabic countries) and 2011 (the Spanish ‘15M Real Democracy, Now!’), there is growing pressure on governments to increase civic participation in the management of society. The traditional democratic model encounters significant difficulties when it is expected to react effectively within complex, uncertain and dynamic environments. The democratic legitimacy of public institutions is being questioned by a better educated, more reflexive and more critical citizenry.

There is a need for new models of participation that can make use of the potential of the Knowledge Society (KS) and respond to the challenges (transparency, participation, control, etc.) that it generates. The determination of a model of participation that is most appropriate for a given epoch is by no means an original topic of debate and discussion; as Ibn Jaldun concluded more than six centuries ago [1], this problem may only be resolved if the dynamic of the system is understood and the system is self-organised and adaptive [2].

Participation must have a practical outcome; citizens must be able to see its impact and results. Joint and in-house evaluation procedures should be developed in order to measure the scope and impact of e-Participation experiences. Evaluation allows the visualisation of the results of an initiative and the degree of achievement in terms of the proposed objectives; it is also an expression of rigour, transparency, analysis and continuous improvement which can reinforce the consistency and credibility of participatory experiences.

Evaluation should aim to undertake the most rigorous analysis possible of the different stages and the results achieved by the e-Participation experiences. It should provide an assessment of the scientific rigour of the methodology: the effectiveness, efficacy and efficiency (EF3-approach), as well as the economic, social and environmental impacts of the actions.

Macintosh and White [3] put forward three arguments for a rigorous evaluation framework: (i) the increasing amount of information available on the internet requires new knowledge and information management systems; (ii) the range of stakeholders demands personalised communication integrated with the delivery of relevant information; and (iii) information systems design must move towards more collaborative working environments to support a partnership of government and civil society.

Aichholzer and Westholm [4] argue that the evaluation of e-participation is indispensable if knowledge of greater precision and objectivity is wanted about the effectiveness, the value, the success of an e-participation project, initiative or programme.

In the last 15 years, a number of e-participation experiences have taken place and many of them have made extensive use of information and communication technologies (ICTs). Some examples and descriptions of e-participation experiences can be found in [5,6,7,8,9]. Many of these projects have made a contribution to the revitalisation of democracy by increasing transparency in governance and creating new political spaces for communication and participation.

Since the emergence of e-Cognocracy as a new model of citizen participation [10,11,12] in 2003, it has been widely studied from a variety of perspectives and viewpoints: political, sociological, scientific, technological and economic [13,14,15,16,17]. However, an integral evaluation of the impact of the application of e-Cognocracy in real-life situations has yet to be undertaken.

This paper presents a theoretical framework for selecting the most appropriate participation model, based on structural equations, which identifies the relevant factors for an integral evaluation of e-Cognocracy; the evaluation simultaneously considers (EF3-approach) its effectiveness (doing what is right), efficacy (achieving goals) and efficiency (doing things correctly). The framework was applied to the evaluation of a real-life practical initiative on the design of cultural and sporting policies for the Municipal Council of Cadrete, Zaragoza. The implementation resulted in the extraction and identification of a series of relationships that allow the advancement of an EF3-participation acceptance model (EF3-PAM), in line with the Technology Acceptance Model (TAM) [18] and the work of Delone and MacLean [19].

The structure of the article is as follows: after this brief introduction, Section 2 deals with the evaluation of e-Participation; Section 3 offers an evaluation of E-Cognocracy and presents the case study; and Section 4 details the main conclusions that can be drawn from the work.

2. The Evaluation of e-Participation

2.1. e-Participation

In the context of governance, participation is a basic democratic right. Sartori [20] describes it as “personally taking part in something; self-activated, willed taking part, not caused by others or mobilised from above”. In general, the term participation includes a range of concepts, from formal participation (exercising the right to vote and the reception of administrative actions) to active participation or political participation in the resolution of conflicts and making public decisions (governance).

The electronic government (e-Government) of society can be defined as the utilisation of ICTs in Public Administration with the aim of improving public services and democratic processes, thereby strengthening support of public policies [21]; e-Government can be applied to the two broad areas of Public Administration activity: (i) the provision of services (e-Administration) and (ii) political participation in democratic processes (e-Governance). Nevertheless, the term e-Participation has, traditionally, been exclusively associated with political participation or e-Governance in its widest sense (participation in public decision making relative to the governance of society).

Given this interpretation of e-Participation (or e-Governance), e-Cognocracy can be considered as a new model of participation which, in the context of e-Governance, provides cognitive collaborative public decision making [22] and addresses [16] the demand of Michels and De Graaf [9] for e-participation models: “…to think about smart ways of combining participatory processes and formal decision-making, for example, by splitting the process into different phases and coupling participatory and deliberative phases to the phases of decision-making.”

2.2. e-Cognocracy

E-Cognocracy [10,11,12,23,24,25] is a new system of democratic representation that combines liberal (representative) democracy and direct (participative) democracy to cognitive ends. It seeks the creation and social diffusion of knowledge and the construction of a more open, transparent, cultured, educated and freer society; a society that is more cohesive and connected, more participative, egalitarian and cooperative.

The new system uses multicriteria decisions as its methodological support, the internet as its communication support and the democratic system as a catalyst for learning. As explained in [16,26], the e-Cognocracy methodology consists of 16 stages grouped in four blocks (Figure 1): (1) Problem Formulation, Stages 1 and 2; (2) Problem Resolution, Stages 3 to 10; (3) Knowledge Extraction and Democratisation, Stages 11 to 14; and (4) Evaluation and Documentation, Stages 15 and 16.

Block 1, Problem Formulation, refers to the initial problem proposed by the representatives or the citizens, its presentation to the actors involved in its resolution and its final setting.

Block 2, Problem Resolution, includes the two voting rounds, in which the scientific resolution of the problem is obtained, using the Internet as communication support and the analytic hierarchy process (AHP) as the multicriteria decision making technique. The rounds are interspersed with an e-discussion process, in which the actors justify their preferences. The online discussion is the step prior to the extraction of knowledge that takes place in third block. The discussion allows the creative capacity of all individuals interested in the resolution of the problem to be incorporated into the decision making process and this accords with Dahl’s proposals [27,28] on the improvement of democracy based on the use of ICTs. The e-discussion allows the active citizens, a minipopulus, to complement the institutions in order to reduce the gap between the representatives/politicians, and the represented/citizens. As argued by Habermas [29], the e-discussion fosters the co-creation of a more cohesive, fair, educated and effective society.

Block 3, Knowledge Democratisation, provides the arguments that support the different positions [14,15], identifies the social leaders (the citizens whose arguments are followed by the majority of the citizens) and shares this knowledge in order to generate individual and social learning.

Finally, and as recommended for any procedure that makes use of public funds, Block 4 analyses the effectiveness (doing what is right), the efficacy (achieving goals) and the efficiency (doing things correctly) of the e-Cognocracy public policy making process.

The methodology for the conjoint design (politicians and citizens) of local public policies comprises the following phases or steps [11,23]: Step1: Project presentation; Step 2: Problem presentation; Step 3: Identification of the actors, factors and alternatives; Step 4: Modelling the problem; Step 5: Evaluation; Step 6: Identification of the initial positions; Step 7: Citizen debate and discussion. Step 8: Evaluation II; Step 9: Identification of new positions; Step 10: System behaviour; Step 11: Assignation of messages to alternatives and justification of positions; Step 12: Evaluation of individual and collective learning; Step 13: Identification of arguments that support the decisions; Step 14: Extraction and diffusion of knowledge; Step 15: Effectiveness of e-Cognocracy; and Step 16: Project documentation (final report).

The new methodology for the evaluation of the behaviour of models of citizen participation in the taking of public decisions uses an EF3 framework that allows the simultaneous evaluation of the model’s efficiency, efficacy and effectiveness.

2.3. The Evaluation of e-Participation

E-participation is still evolving and there have only been a few proposals on evaluation. Bagozzi and Warshaw [30] advanced a secure and stable technology model for predicting users’ acceptance of a range of new technologies that has been widely employed and studied in the last decades. Fred Davis [18] suggested a technology acceptance model (TAM) that explains the process of acceptation of information technology at an individual level. The Davis model is probably the most recognised and utilised in the scientific literature, it is commonly referenced and has been the inspiration behind a number of other, similar, models.

Delone and MacLean [19] designed the ‘Information System Success’ model as a conceptual framework for measuring the complex dependent variable in research on information technology systems. Ten years later, the same authors updated the original model based on changes in the management of information systems [31].

Rowe and Frewer [32] put forward a framework to evaluate participation in general (it was not specific to e-participation); they defined a number of theoretical criteria which are essential for effective public participation and divided them into two types: acceptance criteria and process criteria. Acceptance criteria refer to representativeness, independence, early involvement, influence and transparency that offer a measure of acceptability to the wider public; process criteria refer to resource accessibility, task definition, structured decision making and cost-effectiveness that offer a measure of effectiveness.

Henderson and Henderson [33] constructed a model for evaluating on-line consultations, e-petitions and internet live broadcasting of parliamentary initiatives. There are seven evaluation dimensions and indicators: Effectiveness; Equity; Quality; Efficiency; Appropriateness; Sustainability; and Process.

Macintosh and Whyte [3] developed an evaluation methodology with criteria that cover three perspectives of an e-participation experience: (i) democratic—the overarching democratic criteria addressed by the experience; (ii) project—the identification of the aims and objectives; and (iii) socio-technical—the extent to which the ICTs directly affect the outcomes. Each evaluation perspective is linked to a number of criteria:

• Democratic criteria: representation, engagement, transparency, conflict and consensus, political equality, community control;
• Project criteria: engaging with a wider audience, obtaining better informed opinions, enabling more in-depth consultation, cost-effective analysis of contributions, providing feedback to citizens;
• Socio-technical criteria: social acceptability, usefulness, usability.

In 2009, [4] published an e-Participation model as part of the DEMO-net project, undertaken in cooperation with other European researchers [34]. They reviewed and analysed applied methods for the evaluation of e-participation and gave core criteria and indicators relevant to e-participation activities such as consultation and deliberation.

Mamaqui and Moreno-Jiménez [35] formulated a methodology based on the utilisation of Structural Equation Models (SEM) for the evaluation of the effectiveness of e-Cognocracy.

Luna-Reyes, Gil-García and Romero [36] devised a multidimensional system for measuring and evaluating electronic government. The model seeks to incorporate the approaches currently used to measure electronic government and ideas from published literature related on the issue.

Finally, Wimmer and Bicking [37] gave us the impact evaluation framework, based on evaluation methods of empirical research thereby reflecting the programmatic contexts of the projects. Evaluation is based on the interaction of the elements of a holistic e-participation solution: the participation process; the topics to be discussed; the policy; and the technology and tools employed.

This current paper proposes a theoretical framework based on the three dimensions for evaluating e-participation processes: effectiveness, efficacy and efficiency. The framework is designed for the evaluation of citizen participation experiences and projects, particularly those that involve e-Cognocracy.

3. The Evaluation of e-Cognocracy

3.1. Background

Most researchers, administrators and users of electronic public services agree that the adoption and use of the services depends, to a great extent, on the benefits that are perceived by the citizens. To foment the use of electronic public services by citizens, public administrations need to know the key aspects that encourage their adoption. Given that the online delivery of public services is based on the use of technology, the variables and models of technological utilisation have acquired a fundamental role beyond the factors traditionally employed in the previous literature.

Al-Adawi, Yousafzai and Pallister [38] produced a model that follows the TAM and explains intention to use e-Government websites by postulating four direct determinants: perceived usefulness, perceived ease-of-use, trust, and perceived risk. The TAM offers a promising theoretical foundation for examining the factors contributing to the acceptance of new technologies; it has been successfully applied in customer behaviour, technology take-up, system use and in a variety of studies human behaviour.

As previously mentioned, at the end of the 1980s, Bagozzi and Warshaw [30] created a secure and stable technology model for predicting user acceptance of a range of new technologies. In 1986, in his doctoral thesis, Fred Davis [18,39] defined a technology acceptance model (TAM) that explains the process of acceptance of information technology at an individual level. It is in line with the tradition of previous research into information systems [40,41,42] that aimed to identify the attributes that lead to the success of information systems in business, taking user satisfaction as its measure.

The TAM is based on the Theory of Reasoned Action (TRA) [43] and its methodology of expected values. The authors of the model had already used the TRA in their research. Bagozzi [44] used it in a work on blood donation, Warshaw and Davis [45] in a number of studies and Warshaw [46] in research on brand selection. The TAM permits evaluation measurements of the quality of information technology systems and their adjustment to the requirements of the tasks that are to be executed; it is used to predict the level of acceptance and use of new technologies. The model (Figure 2) assumes that attitudes toward the use of an information system are based on two antecedent variables: (i) Perceived usefulness (PU) and (ii) perceived ease-of-use (PEOU).

This is similar to Bandura’s [47] concept of self-efficacy. Perceived usefulness is defined as “the degree to which a person believes that using a particular system would enhance his or her job performance” [18]. The items employed by Davis [18] as indicators of perceived usefulness are directed at determining people’s evaluations of the consequences that the use of an information system may have on their productivity at work.

Another of the fundamental constructs of the TAM is the perceived ease-of-use of a technology, which is based on the self-efficacy of Bandura [47], defined by Davis [18] as “the degree to which a person believes that using a particular system would be free from effort”. The items that measure this concept are: flexibility; ease-of-use; control; and the simplicity of becoming an expert in its use.

In the TAM, a direct link is made between beliefs (perceived usefulness) and intentions. This is a significant difference with respect to the Theory of Reasoned Action [43], where beliefs only have an impact on attitudes.

In 1992, DeLone and McLean [19] established an Information System Success Model, based on research on communications carried out by Shannon and Weaver [48]. They specified two quality dimensions: System quality, that measures technological success, and, Information quality, that measures semantic success (the content). The model was founded on an analysis of the literature from 1980 to 1990, consisting of around 100 publications. The original model was applied and its results published in more than 300 papers, guaranteeing the applicability of the model.

In 2003, they updated this model to include three dimensions of quality that affect use and user satisfaction: Information quality, system quality and service quality that measure the quality of service delivery. In the updated model, Intention to use refers to attitude; use refers to behaviour (Figure 3).

The two models were utilised for the development of a new approach for the evaluation of e-Cognocracy: the Theoretical EF3-framework.

The Theoretical EF3-Framework

Considering the evaluation of enterprise behaviour [11], the EF3-approach contemplates the following ideas:

(a)

Effectiveness: a political criterion associated with strategic planning or long-term behaviour, related to aspects relevant to the resolution process (doing what is right);

(b)

Efficacy: an administrative criterion concerning tactical planning or medium-term behaviour, related to measuring how well the goals that are set are achieved (achieving goals); and

(c)

Efficiency: an economic criterion linked with operational planning or short-term behaviour that measures the best possible allocation of public resources (doing things correctly).

The theoretical EF3-framework for the integration of effectiveness, efficacy and efficiency can be seen as an extension of the TAM and DeLone and McLean approaches: The perceptions and behaviour of citizens are used to evaluate the processes of citizen participation and the adoption of technology.

The EF3-framework (Figure 4) was designed for the identification of the relevant aspects required for evaluating e-Cognocracy, but it can also be employed in the evaluation of any e-participation model.

In the theoretical EF3-framework: efficiency is “the operational improvement of the current democratic system”; efficacy is the capacity of the current democratic system to “defend the interests of the citizens through their representatives”; and, effectiveness is “the conjoint creation of a better society”.

The relevant aspects determining efficiency are based on the three constructs contemplated by the DeLone and McLean model [31]: the IT application (system quality), the information that is obtained (information quality) and the human resources support (service quality).

Four constructs are considered for the evaluation of efficacy: information, communication, decision and expectations. Information is a unidirectional flow of interaction (usually from the administration to the citizens); communication is understood as two-way interaction: debate and discussion. In addition to the bi-directional flow of information; decision includes the production of a co-decision between the administration and citizens, and expectations refer to the identification of the characteristics that participation experiences should have in the future.

Effectiveness is studied through the analysis of two scenarios as latent intermediate variables: the current situation and the future situation (ideal), and an endogenous variable that captures the impact of e-Cognocracy (the creation of a better society).

The following section of this work presents the application of the theoretical EF3-framework through a survey implemented in the real-life experience of Cadrete (e-Cognocracy) using SEM or the covariance structure analysis approach [49,50,51,52].

3.2. Relevant Aspects: The Estimated Structural Model

The identification of relevant aspects for the EF3 evaluation of e-Cognocracy, using structural equation models, was by means of a survey undertaken after the implementation of the project in Cadrete.

3.2.1. Case Study

In April 2010, the Cadrete Municipal Council, in collaboration with the University of Zaragoza, undertook a citizen participation project aimed at giving the residents of the municipality a voice in public decision making. The project was financed by the Government of Aragon and organised by the Zaragoza Multicriteria Decision Making Group. The issue in question was the design of cultural and sporting policies. There were two main objectives: (i) That decisions on the budget assigned to the aforementioned policies would be conjointly made by the politicians and the citizenry; and (ii) that citizens would be encouraged to involve themselves in the debate and take part in the decision making process and the arguments that supported the decisions would be publicly disseminated.

Participation was encouraged by the incorporation of a new group of actors: the neighbourhood associations. There were therefore three groups of actors that were given different weightings: (i) The politicians, with a weighting of 40%; (ii) the citizens, of 44%; and (iii) the local associations, of 16%.

The participants were local residents (on the electoral register) of over 18 years of age (politicians, citizens and representatives of the local associations). There were two voting options: with National Identity Card or with username and password. As was the case with Mamaqui and Moreno-Jiménez [35], the analytic hierarchy process [53,54] was used as a methodological support and the Internet as the communication support.

The census of actors that fulfilled the requirements necessary to participate and the weights awarded to each group are shown in

Table 1. Voters and Weightings.

Participants	Census	Weight
Associations	15	16
Citizens	1949 *	44
Politicians	11	40
Total	1975	100

* Over 18s with the right to vote in 2008 (data from the Aragon Statistics Institute) (source: the authors).

. The voters could determine what proportion of economic and financial resources should be allocated to each of the four segments of the population considered: children (0–14 years old), the young (15–29 years old), adults (30–64 years old) and the elderly (over 65 years old).

A hierarchy with two criteria (Cultural and Sports) and six sub-criteria was constructed (Figure 5). Within the criterion associated with cultural aspects, three sub-criteria could be selected: Education, Leisure and Identity. Within the sports criterion, the selection sub-criteria were: Entertainment, Physical Development and Social Relations.

3.2.2. Phases of the Process

Following the structure of the e-Cognocracy democracy model [11,12,23,24,25,26], the process of participation was:

• Problem formulation.
• Information and training.
• Modelling the problem: following the methodology of the analytic hierarchy process [53], one of the most widespread multicriteria techniques.
• First round of voting: Those inscribed in the citizen participation census could vote on their (cultural or sports) preferences between 1:00 p.m. and 10:00 p.m., on the 8 April 2010.
• Discussion: From the 8th to the 16th of April, a forum was opened on the Internet. Any citizen, even if they had not logged in for the voting process, could freely express their opinions, proposals, suggestions and ideas, and respond to the comments of the other citizens.
• Second round of voting: As in the first vote, those inscribed in the citizen participation census could vote electronically on their preferences (cultural or sports) from 12:00 midday to 7:00 p.m. on the 16 April 2010.
• Presentation of the results and closing ceremony: This took place on the 23 April 2010. The results were announced and a prize draw was held.

In order to evaluate the experience, at the end of the elicitation process (on the same day and at the same location), the participants were asked to complete a questionnaire (see Appendix A) comprising 51 questions grouped into seven sections: (i) The System of Citizen Participation; (ii) The Creation of a Better Society; (iii) Motivation; (iv) Evaluation of the Technology Support and Applications; (v) Evaluation of the Information; (vi) Evaluation of the Support Personnel; and (vii) Overall Evaluation. The information from the questionnaire was used to analyse the effectiveness of e-Cognocracy.

3.2.3. Methodology

The methodological support used in the voting experiment was the analytic hierarchy process (AHP), a multicriteria technique proposed by T.L. Saaty [53]. First, a hierarchy (Figure 5) that contains the relevant aspects of the problem was constructed by the research team responsible for the Cadrete project. Before eliciting the citizens’ preferences, a brief presentation of the AHP methodology was made to the citizens by the research team. Then, following a top-down procedure, the individuals incorporate their preferences by means of the pairwise comparisons of the elements considered, following Saaty’s fundamental scale (

Table 2. Fundamental scale [53].

Intensity	Definition	Explanation
1	Equal importance	Two activities contribute equally to objective
3	Moderate importance	Experience and trial slightly favour one activity over the other
5	Strong importance	Experience and trial strongly favour one activity over the other
7	Very strong importance	An activity is much more favoured than the other and its dominance is demonstrated in practice
9	Extreme importance	Evidence in favour of an alternative over another has the highest possible order for his claim

). Finally, the methodology aggregates the values throughout the hierarchy to obtain the total priority of each alternative with respect to the objective of the problem [54].

3.2.4. Results

In the first round of voting (

Table 3. The two rounds of voting.

Participants	Census	Electronic Voting in the 1st Voting	Percentage	Electronic Voting in the 2nd Voting	Percentage
Associations	15	3	20%	2	13.3%
Citizen	1949 *	37	1.9%	35	1.8%
Politicians	11	3	27.3%	4	36.7%
Total weighted	1975	43	2.17% (14.96%)	41	2.08% (17.60%)

* Citizens over 18 in the 2008 Census (data from the Aragon Statistics Institute) (source: the authors)

), 43 participants voted electronically; 37 were citizens, three were politicians and there were three from associations. This represented 2.17% of the total census and 14.96% of the weighted participation. In this round, the participants opted for the cultural criteria (52.99%) over sports (47.01%) (

Table 4. Priorities of the criteria of each group of actors.

Criteria		Associations	Citizens	Politicians	Total Vote
1st vote	Cultural	39.33%	57.64%	53.35%	52.99%
	Sports	60.67%	42.36%	46.65%	47.01%
2nd vote	Cultural	54.88%	62.56%	50.47%	56.58%
	Sports	45.12%	37.44%	49.53%	43.42%

(Source: the authors).

). The priorities of both rounds can be seen in [16].

Between the first round and the second round, a forum of debate was opened on the Internet, participants expressed their concerns and preferences (

Table 5. Messages and comments on the Cadrete forum.

	Total Messages	Total Comments	Total
Sport	24	81	105
Culture	37	114	151
Total	61	195	156

(Source: the authors).

). A total of 61 messages were posted, 37 belonged to the cultural criteria and 24 to sports. There were 195 comments about these messages, 114 belonged to the cultural criteria and 81 to sports. In the second round of voting (Table 3), there were 41 participants, 35 were citizens, four were politicians and two were from associations, a weighted participation of 17.60% (2.08% of the total census).

In the two rounds, the variations in absolute terms were minimal, although, due to the small number of voters, the relative variation, at least with respect to politicians and associations, shows significant modifications. The percentage of politicians voting increased (from the first to the second round) by 33.3% and that of associations fell by the same figure (33.3%). The voters again opted for cultural criteria (56.58%) over sports (43.42%), by an increased margin (Table 4).

The low citizen participation in the electronic experience contrasts with the municipal (2007) and the general (2004 and 2008) elections whose levels of participation were 69.9% and 76.6%, respectively (

Table 6. Participation in electoral processes in Cadrete.

Elections	Participation Rate
General Elections 2004	80.2%
Municipal Elections 2007	69.9%
General Elections 2008	76.6%
Electronic consultation 2010	2.08% (17.60% weighted)

(Source: the authors).

).

3.3. Estimated Structural Model

After the e-participation experience in Cadrete, participants were asked to complete an online questionnaire; 24 residents responded though only 20 replies were valid. The questionnaire was considered as invalid if less than 80% of the questions were answered and/or there was zero variability with regards to the total number of questions.

The measurement scale was from 0 to 10 (0 = total disagreement, 10 = total agreement). A total of 51 questions were grouped into seven sections: (i) The System of Citizen Participation; (ii) The Creation of a Better Society; (iii) Motivation; (iv) Evaluation of the Technological Support and Applications; (v) Evaluation of the Information; (vi) Evaluation of the Support Personnel; and, (vii) Overall Evaluation.

The theoretical EF3-framework was first evaluated through a survey in Cadrete using the SEM or the Covariance Structure Analysis approach [49,50,51,52]. The first analysis was a conjoint descriptive analysis of the variables, using the measurements of their position, dispersion and correlation. These measurements led to the identification of groups of interrelated indicators that could be expected to define the utilised constructs; the analysis was complemented by an examination of the main components. The study was completed with structural equation models with latent variables, or covariance structure analysis [51].

This methodological approach was chosen as it allows the researcher to formulate and evaluate the existence of latent variables from the reflected indicators [50], that is to say, variables that are not susceptible to direct observation. The software used was EQS 6.1 [52].

Table 7. Constructs and Indicators.

EFFICACY			*
INFORMATION	X1	The Administration informs society about the existing mechanisms of citizen participation	5.00
	X2	The Administration informs society about the decision taken	4.70
COMMUNICATION	X3	Political powers take citizens’ opinions into account for the design public policies	5.00
	X4	Political powers take associations’ opinions into account for the design public policies	4.95
DECISION	X5	The citizens have weight in political decision making	5.15
	X6	Associations have weight in political decisions making	5.05
EXPECTATIONS	X7	Citizens should participate in the design of public policies	7.50
	X8	Citizens should decide the design of public policies in conjunction with the elected representatives	7.15
EFFICIENCY
INFORMATION QUALITY	X9	In general, I am satisfied with the information that I have received	6.90
SYSTEM QUALITY	X10	In general, I liked the design of the software application	5.95
EFFECTIVENESS
CURRENT SYSTEM	Y1	With the current system of citizen participation, the representatives defend my interests	5.50
FUTURE SYSTEM	Y2	E-Cognocracy improves the current democratic system	7.90
E-COGNOCRACY	Y3	E-Cognocracy contributes to create a better society	7.70

* Average indicator evaluation. (Source: the authors).

shows the constructs and indicators used to describe the effectiveness, efficacy and efficiency in the estimation of the structural model for an EF3-evaluation of e-Cognocracy.

The indicators concerning information, communication and decision, revealed evaluations that were just above or just below an acceptable level of satisfaction (5.0, 4.7, 5.5, 5.0, 5.2 and 5.1, respectively); expectations reached a high level (7.5 and 7.2), which suggests that citizens are not convinced about actively participating in the system. Therefore, efficacy is defined by the indicators information, communication, decision and expectations.

In general, the average valuations of the indicators of efficiency were relatively high. Participants gave a more positive valuation to service quality (8.50) than to information and system quality (7.0 and 6.0, respectively). The correlation was over 0.7, which indicates that the structure should be maintained. Therefore, efficiency is defined by the participants’ satisfaction with the quality of information and of the system. In our empirical study, the third construct (personal support) considered by DeLone and McLean was centred on informing about the AHP methodology, thus it was integrated into the Information Quality.

E-Cognocracy was given a more positive evaluation than the current system of citizen participation (current situation), both in terms of its ability to improve the current system (future situation) and to achieve its ultimate goal-the creation of a better society (7.9 and 7.7 compared to 5.5). It is, therefore, clear that the improvement of the current system and the creation of a better society require the introduction of more dynamic and participative mechanisms, such as those employed by e-Cognocracy. All of the above can be defined as effectiveness. The role of the citizen is perceived as being as relevant as that of the associations. The relationships between the indicators were coherent with the constructs.

The confirmatory factor analysis (

Table 8. Confirmatory factor analysis model.

		INFO.	COM.	DEC.	EXP.	Efficiency	R2
INFORMATION	X1	0.99 ***					0.98
	X2	0.87 ***					0.76
COMMUNICATION	X3		0.95 ***				0.90
	X4		0.98 ***				0.96
DECISION	X5			0.90 ***			0.81
	X6			0.78 ***			0.61
EXPECTATION	X7				0.70 ***		0.49
	X8				0.97 ***		0.94
EFFICIENCY	X9					0.83 ***	0.69
	X10					0.86 ***	0.74
INFORMATION	INF.	1
COMMUNICATION	COM.	0.40 *	1
DECISION	DEC.	0.26*	0.57 **	1
EXPECTATIONS	EXP.	−0.10	−0.04	−0.26	1
EFFICIENCY	Efficiency	0.29 *	−0.37 *	−0.28 *	0.27	1
	C-FL	0.87	0.93	0.71	0.72	0.71
	Omega	0.93	0.97	0.84	0.84	0.85

χ2(25): 26.97, p-value: 0.52; SRMR: 0.07; GFI: 0.90; CFI: 0.99. * significant to 10%, ** significant to 5% and *** significant to 1%. (Source: the authors).

) offered sufficient evidence to maintain a first order structure of four correlated factors (Information, Communication, Decision, Expectation) from the eight observed variables (X1, …, X8). The structures of the constructs, in theoretical terms, were empirically corroborated at both an exploratory and a confirmatory level. The eight indicators reflected a structure of four interrelated latent variables.

Table 8 also shows that the relationships between the dimensions of information, communication and decision were positive and this implies that a higher perception of information signifies a higher perception of communication and decision, and vice versa. The relationship between expectations and decision was negative, and this implies that citizen disappointment with regards to the existence of decision is greater if they have greater expectations of participation. Reliability indices for both these observed variables and their respective latent variables were more than acceptable (R², omega and C-FL).

Figure 6 depicts the estimated structural model.

Table 9. Structural model.

	INFO.	COM.	DEC.	EXP.	Efficiency	CUR.S.	FUT.S.	R2
CURRENT SYSTEM	0.31 *	0.38 **	0.11	0.04	0.50 **			0.58
FUTURE SYSTEM	0.14	0.05	−0.45 *	0.38 *	0.03			0.40
E-COGNOCRACY						0.37 **	0.51 ***	0.41

χ2(46): 43.16, p-value: 0.71; SRMR: 0.07; GFI: 0.90; CFI: 0.99. * significant to 10%, ** significant to 5% and *** significant to 1%. (Source: the authors)

shows the estimated structural model; there are three determinants of the citizens’ perception of the current system: information, communication and efficiency. Determinants of the future situation are decision (negative) and expectations. The current and future situations affect the perception of the creation of a better society, though the effect of the future situation (e-Cognocracy) is greater.

If citizens perceive that information exists, that is to say, that the administration informs society of the participation mechanisms and the decisions that are taken (top-down unidirectional flow), then they have a positive perception of the current system of representation.

If communication exists, that is to say, information flows in both directions (feedback), it is also a determinant, but this is not the case with decision and expectations. Moreover, if citizens feel that they have no influence on the taking of political decisions, irrespective of their perceptions of the existence of information and communication, they will favour a change in the participation system. If the citizens have higher expectations of involvement in the design and formulation of public policies, they will also favour change.

Due to the limited number of responses, it was not possible to validate a general framework for the conjoint evaluation of all the aspects outlined in the theoretical EF3-framework. Nevertheless, the results obtained from the 20 valid responses identified a series of relationships that contribute to the formulation of a general framework.

The small sample size means that the evaluation and selection of the models is governed by goodness of fit indicators (SRMR, GFI and CFI) that do not directly depend on the number of observations [49]. For all the measured and/or structural models, the estimated parameters were presented in their completely standardised version, norm 0–1, and, in addition, all the equations were given their corresponding coefficients of explained variance.

The assessment of the construct is based on the methodology proposed by Bagozzi [55] for the validation of multidimensional constructs and the covariance structure analysis of observed variables (McDonald’s omega coefficient [56] and Fornell and Larcher’s coefficient, C-FL [57]. The stability of the parameters of the models was estimated and evaluated sequentially.

Although the simplified analysis of the theoretical EF3-framework has not allowed significant statistical conclusions, it has meant that, in conjunction with a review of the existing literature, this framework may be extended to evaluate any e-Participation experience, not only e-Cognocracy.

4. Conclusions

The traditional democratic system finds it difficult to efficaciously react in the context of a dynamic, complex and uncertain environment. The democratic legitimacy of public institutions is being questioned by a citizenry that is more and more educated, reflexive, critical and interconnected. Citizens are demanding more open and receptive governments that are prepared to listen, share, and co-decide.

In the search for an appropriate response to the needs of democracy in the epoch of the Knowledge Society, new models of e-participation and systems are being advanced; the validity of these models needs to be analysed by taking into account their effectiveness, efficacy and efficiency (EF3-approach). The integral evaluation of the three perspectives in citizen participation models is still pending; in this work, the EF3-approach has been applied to a real-life experience resolved by means of the cognitive democracy model known as e-Cognocracy. Through the use of structural equations, the relevant factors extracted for the EF3 evaluation of this model will help to analyse the performance of the remaining citizen participation models.

The theoretical model (EF3) was applied to the design of cultural and sporting policies in the Spanish municipality of Cadrete. Due to the limited number of observations, it was not possible to validate a generalised model for the integral evaluation of e-participation experiences. Nevertheless, it was possible to extract a series of relationships and relevant factors that will be considered when dealing with similar cases in the future.

Despite the small number of responses to the evaluation questionnaire, it appears that the experience was seen as very positive by the people of Cadrete. These kinds of projects should be continued; they make a contribution to the conjoint construction of a better society.

Among the relevant aspects derived from the study we can highlight the followings: (i) The efficiency is determined by information and system quality; (ii) The efficacy is explained by information, communication, decision and expectations; and, the effectiveness (creation of a better society) is studied by analysing the change (impact) between the current and future situations.

More specifically, the high value given for the evaluation of expectation is a reflection of the uncertainty that citizens feel with regards to actively participating in the system. The role of the citizen was perceived to be of the same importance as that of the local associations.

The confirmatory factor analysis offered sufficient evidence to maintain a structure of four, interrelated, first order factors based on the eight observed variables. The relationships between the dimensions of information, communication and decision imply that a greater perception of levels of Information will mean that perceptions of communication and decision are higher, and vice versa.

The relationship between participation expectation and decision was negative. This implies that when citizen disappointment regarding the existence of decision increases, there is also an increase in participation expectation. The sample gave a more positive evaluation of e-Cognocracy than the traditional democratic model, both in terms of its ability to improve the present system and to achieve its ultimate aim-the construction of a better society.

As already mentioned, a limitation of the study (that could be resolved with further research) is the small number of responses to the evaluation questionnaire. It is well known that a higher number of responses improves statistical robustness so it is difficult to validate more generalised models. However, one of the strengths of the study is that the analysis was of a real-life situation, and this enhances the internal validity of the research.

Finally, it should be emphasised that the improvement of the system and the creation of a better society require the introduction of more dynamic participation mechanisms such as e-Cognocracy. The ideas extracted in this project for the evaluation of e-Cognocracy could be used for the evaluation of any democracy model that combines, as Michels and De Graff [9] assert, participatory processes and formal decision-making.