Next Article in Journal
Residents’ Sustainable City Evaluation, Satisfaction and Loyalty: Integrating Importance-Performance Analysis and Structural Equation Modelling
Previous Article in Journal
Measures for Ensuring Sustainability during the Current Spreading of Coronaviruses in the Czech Republic
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Impact of Information System on Interactions of Child Welfare Professionals with Managers and Clients

by
Miroslavas Seniutis
1,*,
Donata Petružytė
1,
Miglė Baltrūnaitė
1,
Silvija Vainauskaitė
1 and
Linas Petkevičius
2
1
Institute of Sociology and Social Work, Vilnius University, 3 Universiteto St., LT-01513 Vilnius, Lithuania
2
Institute of Computer Science, Vilnius University, 47 Didlaukio St., LT-08303 Vilnius, Lithuania
*
Author to whom correspondence should be addressed.
Sustainability 2021, 13(12), 6765; https://doi.org/10.3390/su13126765
Submission received: 23 March 2021 / Revised: 3 May 2021 / Accepted: 9 June 2021 / Published: 15 June 2021
(This article belongs to the Section Social Ecology and Sustainability)

Abstract

:
The use of information systems in public child welfare agencies of different countries is challenging primarily for frontline practitioners. This paper unveils how material information system properties are involved in the interactions of child welfare practitioners with their clients and managers. We analyze the properties on their own (functional requirements) and according to the perception of IS users (non-functional requirements). On the basis of the results of systematic literature review and ethnographic study of the use of information system at Lithuanian Child Rights Protection and Adoption Service during the COVID-19 pandemic, we identify the relation between the absence of functionality or its potential failures and data duplication problem. According to our study, data duplication leads to additional time consumption, affects the ecology of the workplace, reduces reliability of information, challenges data protection and finally keeps away practitioners from direct work with clients and exposes them to overcontrol of managers.

1. Introduction

Since the 1980s, human service organizations have employed ever more complex information systems (IS) for recording and processing information about their activity with the clients [1]. Increasing volume of research about IS use in social work since the 1990s [2] has revealed potential benefits of IS, for example: possibility for administrators and practitioners to work more systematically and on the basis of methodically collected knowledge [3]; opportunity to merge measurement processes into the operations of the agency and add structure to the agency’s program [4]; support for decision-making [5]. Nevertheless, well-grounded insights emerged about the fact that far from being only positive, the application of IS can seriously hamper service delivery [6].
A share of studies about use of IS in social service agencies, especially those operating in child welfare (CW) field, have been orientated to micro-organizational interactions. One can make a distinction between studies that focus on employees’ experience of IS use and studies that have given special attention to material properties of IS. The first type of the studies analyzed: (a) capacity of employees to apply gathered information for realization of their work goals [7]; (b) reasons of employees’ dissatisfaction caused by IS failures, problems of access to client information, low quality of information in the system [8], neglection of the needs of frontline workers in IS design [9]; (c) IS breaking effect on the narrative nature of knowledge of practitioners [10,11]; (d) and, in a wider sense, implications of IS for social work practice in children‘s services [12]. Studies of the second type investigated technological failures on their own [13,14]. There were also some attempts to combine these two perspectives, for example, in the studies of workarounds, that examine coping strategies of employees faced with technological failures [15].
Another stream of studies analyzed the consequences of IS use for macro-organizational phenomena. In these studies, significance of application of computerized information management technologies for the growing demand of accountability and for quality assurance was presented [16,17]. However, increasing attention was also paid to the effect of IS on the growing bureaucratization of social work at the expense of practitioners’ direct work with clients [18]. Implementation of IS in CW for evaluation of organization performance and higher transparency in accountability increases organizational control as well, especially in interactions between managers and practitioners. The IS which are taken over from the business world bring along their managerial pattern [19] and instead of helping practitioners and managers to collect and use information systematically, efficiently and effectively [12], quite often IS emphasizes hierarchical organizational structure, where information resources are used to maintain control and authority rather than to respond to the needs of clients or practitioners [20,21].
The aim of this article is to demonstrate how specific material IS properties [14] affect interactions of practitioners with their clients and managers, looking from the perspective of CW professional practitioners. We focus on the material IS properties as technical features that cause challenges in the above-mentioned areas of professionals’ daily work.
The concepts of functional and non-functional IS requirements, taken from the field of social informatics, allow considering material and social aspects of a technological artefact integrally, reconciling the perspectives of user experience and the technology itself and recognising IS as socio-technical system (STS) [22,23]. We use the term of “functional system requirements” to refer to “what” the software performs and “non-functional system requirements” to explain “how well” software accomplishes something [24].
This conceptual frame helps to understand objective IS failures not in an isolated way, but always in a relation to subjective user experience and to professional needs and practices of the users. At the same time, we seek to answer the questions: what changes in interactions of practitioners with clients and managers are brought by material IS? If such changes take place, how their emergence and development may be explained?
We define “professionals” as representatives of different professional areas including social work, who work in CW field and have experience of direct work with clients. We refer to employees in managerial positions as to “managers” in relation to professionals and frontline practitioners, even if managers represent a separate profession.
In the first part of this article, we explain the choice of multi-method approach, especially the connection between the methodology of systematic literature review and the methodology of empirical ethnographic research as well as their relation to the aim of our research. Afterwards, we provide the results of the literature review and present the findings of the empirical research. The novelty of our empirical findings is considered in the discussion part.

2. Methodology

We used a multi-method approach to achieve the aim of our research and to enhance its internal validity [25,26]. This approach involved first-level complementary triangulation, i.e. a type of analysis which uses a phased methodology, where researchers use one method in order to have a background understanding of a different kind of data collection [27]. In our case, we performed systematic literature review in order to build an initial framework related to the following questions: (a) impact of IS on the work practice of CW professionals, (b) specific technical IS features which cause the challenges, and (c) the ways to overcome those challenges. This framework formed the base for the evaluation of the impact of IS on professional practice according to both material and social aspects of this phenomenon [22]. The concepts of functional and non-functional requirements were chosen to indicate the objective material IS properties and subjective IS user perception of these properties that discloses the problematic aspects of IS use for practitioners’ interactions with clients and managers. Based on this framework, we formulated the main goal and the instruments of our empirical research.
Alongside with the validation of the method, we tried to enrich our analysis and achieve a more complete understanding of our research topic by applying a second level of complementary triangulation [27,28]. During a complex empirical research carried out in 2020 at Lithuanian Child Rights Protection and Adoption Service (CRPAS), we used mixed method approach by combining quantitative and qualitative data collection methods [29]. First, we conducted interviews with CW professionals to review our initial framework constructed during systematic literature review. Secondly, we conducted participant observation of daily work practice of CW professionals in order to verify that verbal accounts and practices correspond to each other. Afterwards, we elaborated a questionnaire based on the results of qualitative data analysis in order to evaluate the extent of emerged concerns about the problematic impact of IS. The questionnaire took into account different points of view of staff members. The fact that we use all the three methods reflects our intention to realise triangulation as well as to converge the outputs from different data collection methods [27].

2.1. Identifying Need for Literature Review

Our search for existing literature reviews of studies about the impact of information systems on practice of CW professionals did not yield any positive results. Several literature reviews dealt with related topics only. Liedgren et al. [30] analyzed the impact of technological decision-making tools in a broad sense on decisions of social work professionals in different fields without giving special attention to CW field. Saxena et al. [31] analyzed the impact of IS upon CW specialists with a special emphasis on algorithms which support evidence-based activities of practitioners, but problems of IS functionality were little addressed.

2.1.1. Formulation of Research Questions

In order to get more systematic understanding about the effect of IS on CW professionals’ practice, we elaborated our research questions adopted form software engineering field [32]. The first question is aimed at identifying the impact of technology on the performance of professional practice, the second one at identifying the risk factors associated with technology, the third one at identifying the strategies of coping with technology failures
Q1.
What impact does the IS have on daily practice of CW professionals?
Q2.
What technical IS features (functionality, architecture, design, etc.) inhibit CW professionals from pursuing their professional goals appropriately?
Q3.
How are the problems related to IS use in the professional practice of CW professionals addressed?

2.1.2. Systematic Literature Review

The protocol of our systematic literature review follows Kitchenham’s and Charters’s [33] guidelines for software engineering researchers. These guidelines were created as an instrument for systematic literature review of various disciplines. This was a crucial factor for our interdisciplinary research which focuses on the impact of IS on CW professionals’ practice. The identification, screening of systematic review was done in August–September of 2020, while the systematic review synthesis was done in October.

2.1.3. Selection of Academic Research Databases

According to systematic literature review guidelines [33], we selected the following academic research databases for literature review:
(a)
Clarivate Analytics Web of Science;
(b)
ACM Digital Library.
Initially, we also considered databases Google Scholar, Scopus, IEEE Xplore Digital Library and Springer Link However, initial queries showed that Google Scholar engines yielded a huge amount of results where most of quality research papers overlapped and were indexed in Clarivate Analytics Web of Science and majority of them were presented in the form of conference proceedings which automatically could not be included in systematic literature review. After analyzing initial queries, we concluded that additional databases did not provide satisfying search results nor allow strict queries or their results overlapped with those of selected databases. The selection flow is presented see Figure 1.

2.1.4. Definition of Selection Criteria

On the basis of above-defined research questions, we established the following selection criteria (see Table 1): The period 2010-2020 was selected in order to include the latest and the most relevant research studies. In addition, until 2010, “little attention has been paid to how these systems [IS] impact human service practice“ [34] and the research on the subject proliferated only afterwards, particularly about the problematic aspects of IS use. We selected for English and Lithuanian language papers for the review, the first language due to its prominence in scientific research and the second one with the intention to examine contribution of national researchers on the subject. We performed the search according to the defined inclusion criteria in the title, the abstract, the introduction and the conclusions of each paper. Four researchers have applied the selection criteria to randomly chosen group of articles (each article was screened by one reviewer). The doubts about the attribution of inclusion criteria were discussed in the group of researchers.

2.1.5. Definition of Quality Criteria

In order to reduce possible bias and increase validity of selected studies, we defined the following quality criteria (QC):
QC1.
Study is indexed in Clarivate Analytics Web of Science database and ACM Digital Library.
QC2.
Study deals with the impact of IS on professional practice of specialists working in CW field or with development processes or methods of IS that support their work practice.
QC3.
Study results are based on empirical research.
QC4.
Study provides recommendations how to improve CW professionals’ practice by using IS or what aspects of IS functionality should be improved and how to do it in the most efficient way.
QC5.
Study presents results that are relevant for assessment of impact of IS on professional practice of CW professionals in State Child Rights Protection and Adoption Service of Lithuania and for development of supplementary functionality of IS used in this institution.
According to guidelines of structural literature review, we assigned each source included in the review with a score equal to number of quality criteria met by a source. We considered that the source met a criterion only if it completely fulfilled its conditions. We considered that a source with a higher quality score was more relevant and more helpful for tackling research questions.

2.1.6. Construction of Search Strings

On the basis of specified research questions, we constructed search strings that enabled the results of the review to be reproduced and validated by other parties, and that used several logical operators in order to increase the number of potentially relevant sources. The queries that we used are presented in Table 2.
Some of studies selected for systematic literature review focused on IS use in work practice of social workers. Meanwhile, the object of the review that we have performed covered work practice of professionals of different occupations (psychologists, lawyers, educators, etc.) including social workers who work in CW field. A narrower occupational focus in selected studies did not contradict the goal and the objectives of our review but permitted grasping their different connotations.

2.1.7. Selection of Data Sources

During the conducting phase (see Figure 1), we processed selected data sources with the reference management tool “Mendeley”. We categorized sources according to their academic research database. After categorization, we removed duplicate entries and thus we selected 49 articles for the next stage of literature review.

2.1.8. Data Source Quality Evaluation

We evaluated selected articles according to the above-mentioned quality criteria, assigning a score from 0 (when criterion conditions were not fully met) or 1 (when criterion conditions were fully met) for each criterion. Thus, each data source received a score from the continuous interval [0; 5]. A source was admitted into analytical phase only if it had met all the five criteria fully (the score was 5 out of 5). After quality evaluation of the articles, we selected 13 articles for systematic literature review. Assessment scores of all the studies are provided in Appendix A Table A1.

2.1.9. Data Synthesis

We summarised the results of the included studies by using descriptive (narrative) synthesis method dedicated to the analysis of qualitative data. According to the line of argument synthesis, the researchers infer their conclusions about a topic as a whole from a set of selective studies that look at a part of the issue [33].

2.2. Empirical Research

2.2.1. Research Design

We chose ethnographic research design as widely approved by scientific community that focuses both on impact of IS on work practice in business and human service organizations and on designing new forms of IS [11]. The chosen epistemological approach is centered on observation of activities situated in their natural settings and on discussions with experienced practitioners about their routine activities. This approach has given us opportunity to retrace impact of non-human factors such as technological tools involved and actively participating in these activities [35]. As we have placed the focus on the face-to-face interactions between CW professionals and IS, our research strategy was closer to micro-ethnographic one [36], while at the same time we kept in mind broader organizational context in which interactions took place.
A large part of studies from social informatics field [37] which examine the role of information technology (IT) in social and organizational changes as well as the ways that the social organization of IT is affected by social forces and social practices, underline the importance of interdisciplinary approach as an effort to unify different scientific skills by proceeding with generation of knowledge about use of IT in social contexts [38]. In our research group, we combined sociological and informatics competences and also included the views of both practitioners who deal directly with social problems and the IS administrators during the research data collection phase, thus making our research interdisciplinary.

2.2.2. Data Collection

Our methods of data collection included participant observation, in-depth interviews and questionnaire survey. In this paper, we give particular attention to data collected in interviews and we use data from field notes and questionnaires for contextualization of analyzed interviews.
Initially, we conducted 6 semi-structured pilot interviews with employees of different positions: managers, practitioners, government officials, and IT administrators in order to construct and test the interview instrument. Afterwards, we carried out 15 in-depth interviews with frontline practitioners. Simultaneously, we conducted 1-month participatory observation at one child protection office, observing daily work practice of frontline practitioners from 3 different staff teams with particular focus on their interactions with IS. On the basis of the results of qualitative data analysis, we elaborated questionnaire survey intended for all employees of the organization (n ≈ 700), we tested it and carried it out (n = 242), seeking to understand the extent of the raised concerns.

2.2.3. Data Analysis

Using qualitative content analysis method, we performed analysis of qualitative data sources, both of participants’ oral accounts, elicited in interviews, and of field notes that took into account observed patterns of social interactions. We chose inductive category development strategy for definition of analytical units and construction of category system. We reviewed coding scheme and discussed dimensions, definitions, and discrepancies of the categories [39]. We used Maxqda 2020 software as a tool for organizing the material for different analysis procedures. We used R 4.0.2. software package for the analysis of quantitative survey data and for the visualizations of descriptive statistics. In this paper, we present nominal variables with mean, median and interquartile range (IQR) indicators.

3. Results of Systematic Literature Review

3.1. Answer to Research Question Q1

Our first research question of the literature review is the following: What impact does IS have on daily practice of CW professionals? In this stage of the review, we focus on changes in practice of professionals that occur due to IS use on both micro and macro levels of organization. Further below, we summarize relevant findings of other researchers, grouping them by their main subject.

3.1.1. Assumptions and Consequences of Time Consumption for IS Usage

Despite existing evidence that access to reliable internet connection and digital databases should guarantee flawless processes of data collection, management and dissemination [40], most of researchers mentioned quite a few reasons why IS happen to be time-consuming.
First of all, it takes time to understand and learn out how IS works and this applies especially to new employees. Secondly, practitioners spend their time for verifying the information which may be inaccurate, missing or incorrect. One of the reasons behind inaccuracy, ill-timing and fragmentation of data entries may be the fact that social work practitioners use technology for their work tasks only because they feel obliged to do so [41]. These shortcomings of information may lead to issues of trust in IS [42]. Thirdly, data collection by hand-written reports [17] and search for information about client in IS [40] also may lead to additional time consumption. When frontline practitioners are assigned with the task of entering data into IS or other administrative responsibilities, they are distracted from their direct face-to-face work with the clients and their direct involvement in provision of services [43]. It appears that application of IS increases bureaucracy in an organization and gives an emphasis on accountability for services rather than on their improvement [44], even if not all organizations, due to lack of time resources are disposed to implement IS [45]. The speed of responses to client requests also depends on the lack of shared information systems between different institutions, departments, and practice fields [15,46] provided that confidentiality of clients is not compromised [47].
The above-mentioned studies revealed that involvement of CW professionals in different data collection, processing and management procedures related to the functioning of IS does not help saving working time of practitioners. Individual factors that make IS use a time-consuming process, may be identified as follows: ability/inability to use IS, motivation/demotivation arising from obligation to use IS, confidence/distrust in IS. There are also systemic factors, such as increased error-likelihood due to paper-data collection tools embedded in IS or its certain functions, which may increase bureaucracy and reduce time allocated for direct work with clients.

3.1.2. Shaping Discourse about Client

Information stored in IS has characteristics that may have consequences on its application. Sometimes, case reports can be very fragmented [40]. What is more, data available from IS may be insufficient for capturing a realistic view of clients and for creating possible responses to their needs [44]. There may be a lot of information available on IS, but it does not necessarily mean that a wide range of this information will be processed and used for work. Usually, practitioners tend to limit themselves to a quick information review [40]. Whatever detailed and extensive original information about clients may be, its records are limited and shaped by intrinsic material properties of IS [1]: only information which was entered on IS has a value [43]. As IS restricts the type of information that may be entered in it, it also has an effect on contents of discourse about client [44].
Therefore, specific information characteristics have an impact on the narrative content about client. This may shape the way that the client is perceived by CW professional. Therefore, the way of understanding clients and attitudes developed towards them depends on available data which are affected by IS. It is debatable whether an access to fragmented, limited, insufficient or excessive information, which is of quantitative nature in most of the cases, may help practitioners to respond to clients’ needs and preferences in humane and holistic ways [48].

3.1.3. Changes in Practitioners’ Knowledge and Decision-Making

As the researchers put it, the nature of knowledge used by practitioners is increasingly changing from ‘social’ to ‘informational’ [1]. Standardised descriptive knowledge which is provided by IS raises challenges to expertise of social workers and their causative knowledge behind the situation and reduces openness of their work [44,49]. For example, incorporating risk assessment tools in IS may impede professional development of new staff and reduce creativity of experienced staff. It should be noted that standardisation and formalisation of professional practice by introduction of IS-related decision-making tools or time restrictions imposed on work tasks increase probability of errors [43]. In the opinion of practitioners, standardised indications of IS for risk evaluation happen to be not clear enough and ‘fairly abstract’, ‘too general’, ‘not specific enough’, and therefore workers opt for not entering probable risks into IS. Obligation to inform the parents about inclusion of identified risks in IS may complicate relations with parents and hamper the process of helping a child [47].
According to many researchers, quantitative, fact-describing information reduces need to use CW professionals’ causative knowledge and, in this way, it threatens expertise, creativity and professional development of practitioners. IS-related decision-making tools simply standardise both information and practitioner’s responsive actions and do not take into account specific situations, creating additional errors and tensions between practitioner and client. Therefore, information recorded and used in IS may have its effect not only on discourse about client but also on knowledge and decisions of practitioners.

3.1.4. IS as Organizational Performance-Measuring Tool

Usually IS is involved not only in horizontal employee-client type interactions, but also in vertical interactions between subordinate and leading staff: frontline practitioners and team leaders, team leaders and senior managers, organization’s administration and government bodies. Quite frequently, motivation behind IS implementation in organization is not driven by expectations of service improvement but simply by wish to satisfy the funding bodies and by increasing pressure of performance measurement [45]. Similar rationale of manager satisfaction may be observed in lower levels of subordination. In order to meet requirements of organizational managers to record data strictly according to structure and vocabulary of IS tools, practitioners tend to modify initial information about work with client and adapt input to standards of the software or paper-based data collection tool [1]. IS may raise some ethical issues in organization if it becomes a tool in hands of higher-level managers for controlling and monitoring activities of frontline practitioners, for example, seeking to ensure their compliance with standard practices or claiming responsibility of practitioners for their actions in critical situations [43].
In the cases where IS implementation does not aim to improve services but is related rather to accountability at different levels of subordination, employees in subordinate positions are confronted with IS as a tool of control by their managers.
Analysis of results of other research studies revealed that, first of all, IS has its bureaucratic effect which keeps back the goals and practical activities of both organization as a whole and diverts individual CW professionals from their direct work with the clients. Secondly, information stored in IS is inevitably standardized what makes the discourse about the client more fragmented. Thirdly, IS may play its role in standardizing and formalizing the decisions, thus restricting the use of practitioners’ expertise and limiting their activities in situations that need individualized solutions. Finally, control function of IS may lead to setting the goals at both micro and macro levels that are related more to ambition of managers to meet measurable performance indicators rather than to service improvement.

3.2. Answer to Research Question Q2

The main question of this stage of systematic literature analysis is the following: what technical IS features (functionality, architecture, design, etc.) inhibit CW professionals from pursuing their professional goals appropriately? We would like to keep an analytical distinction between changes that occur in work practice of CW professionals due to IS use and due to material properties of IS which have their effect on work practice. At the same time, it is impossible to consider IS functionality without linking it to everyday practice of CW professionals, while changes in work practice stay attached to specific technical IS features.

3.2.1. Technical IS Problems

Although IS should be simple and easy to use [40], in reality they happen to be slow, complex and non-functional. For example, IS accessibility problems arise when remote access to a single database extends user interface time [44]. Accessing and using the system also becomes problematic when some functions or buttons in IS stay unclear (especially to new employees) and some functions necessary for work are missing or incomplete, or require multiple steps for saving or using documents [42].
Lack of necessary functions or their malfunctioning quite often means limited IS use what hampers timely execution of tasks [40,44]. It also leads to multiple additional practices like necessity to have additional local platforms where the information can be saved (ex. Word, Excel) [42] what raises another common/usual problem of duplication of data input. In addition, missing links among different modules of data storage may raise issues for data search [15]. The above-mentioned technical problems oblige practitioners to make a lot of clicks and increase their navigation time in the system what could be pointed out as design flaws.
Connecting different IS technical issues with their effects on the practice of CW professionals, one can distinguish the problems of data access, storage and search, which are the most acute for practitioners involved in data management. Restricted functionality of access to IS and absence of necessary functions force CW professionals to search for alternative solutions of data storage by duplicating them. This process is closely related to lower possibility of finding necessary data.

3.2.2. Mismatch between IS Design and Needs of Practitioners

According to reviewed research studies, the most common reason behind technical issues is lack of collaboration among IT system designers, companies that provide IS, organization managers, practitioners and other players involved in development of IS. The attempts to put particular views, activities and priorities of end users into a technical tool are rather problematic as designers may be inclined to put their own values and meanings or views of senior managers into IS without adapting them to specific conditions of the workplace and to the needs of practitioners [41,42,43]. Positions and interests of other external stakeholders are no less important to the process of IS development. For example, absence of clear government leadership may lead to confusion of the definition and the objectives of a particular IS, and to complicated processes and procedures of data collection [17] reducing practitioners’ capability to detect child protection incidents [15]. The reason for recording particular information should be made explicit in the design of IS to avoid wasting human resources on gathering excess information [44].
In summary, interests of different stakeholders involved in the process of IS design have their effect on the purpose, the objectives, the structure, the processes and the procedures of the system under development. IS functionality is less problematic and meets the interests of all the stakeholders only if their equal participation in IS design process is ensured, especially taking into account the needs of practitioners who work directly with clients.

3.2.3. Restriction of Client Profile Caused by IS

One more highly debated issue is the effect that material IS properties may have on characteristics of information itself. One should admit that a practitioner himself may have some effect on this process by intentional or unintentional selection and filtering of information included in IS records with an emphasis usually made on positive rather than negative information [49]. As one can see, IS is not a passive tool only but it can have its own effect on information characteristics. First of all, categories formulated in IS designing stage, imperceptibly change meaning of information and affect the way how people understand it and react to it [43]. Information in an IS is covered with certain expressions and structured in such a way that it acquires a certain chronological perspective, what affects client’s history record in IS as well as individual workers’ linguistic and contextual choices in recording process [15]. IS designers build into technological tools certain interpretive schemes known as “embodied structures”, for example fields or windows with limited number of words [1] or even predicted number of category choices that can be entered into a file, but that do not provide enough of details about the real situation. There are some cases noted by other researchers when it is not possible to report a risk if it is different than are those already formulated in IS [15].
To put it in another way, categories and linguistic expressions defined in IS designing process and used for linking together and managing the whole set of information, change the meaning of information and alters its original perception. Reduced number of category choices or fields, which may contain only a limited number of words, univocally contribute to sorting-out and fragmentation of information as well as restricts its quantity and content, what has a crucial effect on decisions taken by practitioners, sometimes including their possibility to initiate process of assistance.
Our research overview revealed the most common IS technical problems that are especially related to the functionality of accessibility, search and data storage. The source of technical dysfunctions was successfully traced back to IS design development phase where different stakeholders are promoting the functions that correspond to their own interests and, in this way, define structural framework of future IS. Finally, IS design elements were discussed, which structure information about client and define the way of collecting, understanding and using it.

3.3. Answer to Research Question Q3

Our third research question is the following: how are the problems related to negative IS impact on work practice of CW professionals addressed? In this stage of review, we provide a summary of suggestions of other studies of possible responses to challenges related to IS use.

3.3.1. Use of Technological Tools

Researchers also emphasized importance of using technological tools, when working with IS, in order to save time, to avoid double work, and to reach more flexibility concerning working place. Use of digital tools (for example, tablets and digital pens) can support note-taking. As Huuskonen and Vakkari [49] stated, digital notes are more legible than the handwritten ones and they can be directly put in IS. Consequently, this might diminish work duplication and make recording process faster. Availability of emerging mobile technologies may enhance ability of frontline practitioners to connect to IS also out of office and to access information about service users easily and quickly [41].

3.3.2. Distinguishing the Functions of Specialists

When discussing how to resolve the problems related to negative IS impact on work practice of CW professionals, researchers advise distinguishing between specialists (in larger or smaller teams) who work primarily with clients and those who work with IS. This distinction of special job functions must be made according to interests and abilities of particular specialists. It would help to improve the use of employees’ competences, diminish conflict between direct work with client and IS, and improve recording of client information [44]. Another way to implement this distinction could be working in pairs, when one worker focuses on direct work with client and the other worker may focus on note-making [49].

3.3.3. Client Involvement

Another strategy, which, according to researchers, could help to resolve problems related to negative IS impact on work practice is involving clients in data collection by providing them access to their files and also encouraging them to contribute to data collection [44]. Clients ought to have a parallel access to IS to directly record their impressions there. This opportunity would increase clients’ participation and trust in a client-worker relationship but it would also provide an alternative and differently selected account [49] as well as, in some cases, improve economy of time.

3.3.4. Toolkits and Training

Finally, learning IS use would be made easier by supplying it with instructions, toolkits and training. The problems related to IS use by accidental users (new, inexperienced staff), can be minimized by providing instructions within IS [41] or by creating illustrative toolkit for workers and also toolkit for high- and mid-level managers planning IS. Practitioners do not need an exhaustive mapping of IS, but only illustrative and relevant examples [48]. Lecluijze et al. [47] recommended introducing not only manuals, but training sessions too.
First of all, information and communication technologies connected to IS over mobile network may support data access and procedures of data collection and search. Secondly, working in pairs or separating between IS administrators and practitioners working directly with clients would spare the latter ones of procedures of data collection and entering, thus reducing their workload and the probability of errors. Thirdly, involving clients into data collection process by giving them access to IS would also contribute to reduction of administrative work of practitioners with IS. Finally, automated instructions would help CW professionals to get used to IS and integrate the possibilities it provides into their professional practice.

4. Results of Empirical Research

In this part of the paper, we present those results of our empirical research which confirm, complement and deepen the expertise of other researchers summarized in systematic literature review.

4.1. Serving IS or Client?

4.1.1. Working Time Allocation

Research participants expressed their dissatisfaction about the fact that instead of spending their time for direct work with client, they spend entire hours for filling in different documents: “In reality, we work more with the papers than with the family. Really, it would be good to give more [time] to the family” (D1 (We coded the participants of the interviews by codes from D1 to D17 according to their interview number.)). The survey data revealed that CRPAS professionals (except for managers and administration staff) actually allocate the lion’s share of their working time for computer work, largely at the cost of direct work with client (see Table 3) (8 h = 100%.) (“+” means that professionals allocate more time for an activity that they would like to; “−” means that professionals allocate less time for an activity that they would like to).

4.1.2. Data Duplication

Duplicate information in different paper-based data collection tools. Duplication of information starts in data collection phase already. Professionals use different paper-based data collection tools where a part of information about client is repeated: “I would like to talk to one more child in the meantime [...]. And I have to waste my time for filling in two identical documents” (D9).
Multiple procedures of data collection. Data duplication is caused also by multiple procedures of entering the data. Despite the requirement to enter information about client directly into proper forms which should be immediately loaded into institutional information system (IIS) (document management system), professionals fill in information by hand into draft forms in order to avoid errors which may occur due to different disruptions: “[when you visit a client] it is not always convenient, it is not always that you have some place to sit down, it depends on the situation, sometimes you may have some dogs who stand and bark, you see. You are not able to concentrate and write down in the way that, the questionnaire, you would put into [IIS] under your signature. So sometimes you write it down, and afterwards, oh, here I could use some other word, here I could put it differently, I could change the order of the words, well, such [thoughts], you see, because you are really distracted at the moment” (D1). Afterwards, hand-written notes are transferred to official form in the computer, then electronic document is printed out and professionals store its paper copy in their personal archives in order to ensure access to the document in case if it cannot be retraced in IIS. Digital version of the document stored on the hard disk of the computer is additionally uploaded to IIS. Finally, professionals aggregate a share of data about the case from different records stored in IIS and enter it in government information system (GIS). Thus, in a more or less automated way initial information about client is repeated five times at least and stored in at least five storage places: paper draft, electronic document, printed-out paper version of the electronic document and electronic document uploaded to IIS, while aggregated data are entered into GIS. Data rewriting leads to additional data storages and vice versa, absence of a single platform for data storage multiplies data collection procedures and leads to data duplication.
Lack of connection between separate data storages. It is noteworthy that two separate information systems: IIS, which is used for managing all internal documents, including case records, and GIS, which contains only aggregated information about work with client, are not sufficiently interconnected what does not contribute to reduction of data duplication and hampers search of client information what distracts practitioners from fulfilling their tasks of direct work with client. “For example, there is one information [in IIS], which has not been entered [in GIS], or the contrary case, that really happens. This is why I check in both places [...] That would be great if there existed one, clear and defined system, where you can find everything, but I do not have such illusions” (D10).
Survey data showed that using maladapted paper-based data collection tools is more time-consuming as compared to other issues of data duplication. Professionals have scored activities that take the most of their working time, from 1 to 7, assigning 1 to the least time-consuming activities and 7 to the most time-consuming activities. Priorities of activities of professionals (in this case, we counted only the answers of professionals who investigate reports about possible violations of child rights) were as follows: in the first place (6.75)—“I fill in questionnaires which are too long”; in the second place (6.69)—“I fill in questionnaires which are too many”; in the third place (6.67)—“I fill in questionnaires which contain duplicate information”; in the fourth place (6.18)—“I upload additionally or register in IIS hand-written or computer-typed questionnaires”; in the fifth place (5.31)—“I enter the information about client in two separate information systems”; in the sixth place (4.77)—“I search information about client in two separate information systems”.

4.2. Problems with Functional and Non-Functional IS Requirements

Information systems, like any software, are designed and created by taking into consideration functional and non-functional requirements. While formally many functional requirements are implemented, but in many cases, they are not necessarily practical, or assuring good user experience. The related problems that we have identified are discussed below.

4.2.1. User Load Balance Support Issue

Survey participants expressed their dissatisfaction with very slow operation of IIS while explaining that collapses of functionality due to high number of connected users led to repetition of performed actions. “If there are a lot connected people, it is clear that it works slowly, and especially during the period of quarantine […] you attach, you click for registration and everything disappears. And again, you collect all the data, you attach again, you search” (D1). The disruptions of GIS functionality occur much less frequently and usually are caused by system updates. This is reflected in the answers of survey participants to the question which technical problems CRPAS professionals face the most often. It appears that IIS malfunctions by 33% more often as compared with GIS (see Figure 2).

4.2.2. Strict Restrictions of Data Access

One function which protects data on the one hand but on the other hand aggravates convenient access to information stored in IS is strictly limited time of access session. In the case of GIS, it is very short and therefore it disturbs the work of professionals as they have constantly to log in anew: “Well, let’s say, you log in, don’t you, you connect, if you turn away for a minute, that’s it, you have to [connect] anew again” (D1). This problem has become critical with the COVID-19 pandemic, when usage of systems increased.
Despite the fact that it is possible to access data stored in GIS using devices with mobile internet connection, professionals avoid doing this as design of the webpage is not adapted to smart devices. Meanwhile, access to IIS, where professionals receive their work tasks, is limited to their office. The two systems restrict the remote data access due to their lack of connection to mobile application which could support the practice of professionals. It is especially important when one has to react to urgent cases and there is no time to access data from the office. This situation also restricts possibility of practitioners to receive information about emergency situation and react to it in time (when working out of office).

4.2.3. Limited Design of Database Data Search

The main drawbacks of search function in IIS are: (a) limitations in structural search options within IS: “I cannot search by address, you see, I cannot find the family using surnames of the parents because we register everything under the name of the child” (D10); (b) raw data search without semantic and similarity weighting which returns a large number of uninformative results. Quite often, the system returns accidental registration numbers of the searched documents. The survey participant has compared the IIS search function to “Google search when you enter no-matter-what and you get no-matter-what. That is, whatever you enter, you will receive a heap of something hazy and have to select in that, excuse me, tangled mess, what you need. Absolutely, that does not function. Completely” (D10). The main problem of information search in GIS is inaccurate data entered by employees.

4.2.4. Poor Design of Database Information Filling

In some cases, it is professionals themselves who make mistakes in documents which they fill in and later on upload to IS: “You put in everything by hand: the name, the surname of the child. We are human, we make mistakes and if I enter by accident two letters “A” in “Aleksandras”, never again in my lifetime will I find this child [in IIS]” (D10). In other cases, employees of other institutions provide erroneous information what makes it impossible to find client in IS: “for example, there comes a notification from police, and there is a mistake, for example, in the name, one letter or so. You enter [the data] from that notice [to GIS], but there is no [such person over there]” (D1). The informants expressed their concerns about errors which occur while connecting documents in IS: “it is risky to have such [IIS] when one specialist is responsible for [entering the data] […] if I make accurate and appropriate connections, my colleague will find it when he will need the information in half a year, but if somewhere my human factor breaks down and I don’t do something correctly, he will never find that document, you understand?” (D10). Error monitoring system could reduce risk of inaccurate data records and help saving time in searching information. In addition, application of such mechanisms would contribute to standardization of procedures and information entries.

4.2.5. Challenges of Connecting Documents

It appears that not only human factors but also technological aspects, namely the function of linking different records, is an important factor behind inaccuracies which appear in interconnections of documents. From the perspective of a professional, connecting documents means “waiting a long time” (D1) and having a fast reaction (whatever contradictory it may sound). The documents in IIS are interconnected by using their identification number. “For connecting, I have to enter the number [of initial document], don’t I, and then I have many many many many many little numbers which start blinking and they are blinking so very fast, tiny tiny letters, that I have to catch with the mouse the one I need [that of initial document] so right away right away. Frequently you catch the wrong one, you don’t notice that and the document gets connected with absolutely another case record. […] I even don’t speak about the colleagues who, well, are not dreadfully attentive […] then it’s a complete nightmare” (D10). In order to ensure the retraceability of document in case of wrong connection, professionals take supplementary action and indicate themselves as additional recipients of connected document. One may have an impression that connecting documents is like participating in a competition between IS and its user where one competes to show who is smarter and faster.
Empirical data leads to distinguishing two types of problems which increase time consumption of CRPAS professionals, at the same time distracting them from direct work with the clients. First, it is duplication of data about client due to: (a) superfluous quantity of paper-based data collection tools, their stretched-out and repetitive contents; (b) multiplication of data collection procedures; (c) multiplication of data storages. Secondly, it is the technical issues: (a) slow operation and functionality failures due to surplus number of connected users; (d) lack of mobile application which would have a connection to IS and would grant a remote access to data (or even a possibility of remote data entering); (e) restrictive search parameters and inexplicit search results; (f) lack of automated error and potential anomalies’ monitoring mechanisms; (g) user interface maladapted to data connection.
Moreover, all of these troubles increase staff mistrust in IS, thus leading to proliferation of backup document copies and worsening of data duplication problem what does not contribute to creation of ecological working environment.

4.3. “Client = Offender”: Objective Information or IS-Shaped Attitude towards the Client

In systematic literature review, we have extensively discussed characteristics of information stored in IS and their effect on discourse about client as well as on professional knowledge and decision-making of CW practitioners. Oral accounts of Lithuanian CRPAS professionals during interviews allowed us relating the importance of entirety of information stored in IS to CW professional’s attitude towards the client. Interview analysis allowed us to summarize in a bulleted list below opinions of informants about client information available in GIS and IIS.
(a)
Information helped them in decision-making: “When you go to visit a family you already have a lot [of information] and you know about that family. And thus, after that, well, you really are able to make some relevant decisions, when you have a lot of information” (D1).
(b)
Information was sufficient to receive an accurate immediate impression about client and event circumstances: “one can say that [based on data of GIS] you can have a picture [of the situation] which, I would say, largely corresponds to reality” (D1).
We asked CRPAS professionals to evaluate the level of detail of client’s picture that one can make based on information from GIS and IIS, by assigning a rating from 0 to 10 where 0 meant “available information is completely insufficient” and 10 “available information is completely sufficient” to reproduce a detailed picture. The average rating of information available in GIS stood at 6.443, its median made up 7 and IQR (5, 8). Respective indicators for information available in IIS were 5.943, 6 and (5, 8). Both estimates are significantly different from 5. Therefore, we may conclude that, based on opinion of respondents, available information was sufficient for reproducing a rather detailed picture about client (see Figure 3).
(c)
Nonetheless, information could not be fully trusted due to errors, which at least partially occur due to duplication of data entry procedures: “because there are really such human […] errors, aren’t they, that for example, one information is in Integrra but it is not available on SPIS, or vice versa, that really happens” (D10).
(d)
In addition, information showed only a narrow set of client characteristics. We asked the professionals what kind of client’s picture may be reconstructed from information available on GIS, and several distinct informants noted that the entirety of available information pictures a client as an offender: “[a client] is an offender of child rights. But we do not even collect other data” (D9). The informants explained that paper-based data collection tools even do not provide possibility to collect additional and more diversified information about client: “the picture [of the client] is very negative which does not always correspond to reality because I simply do not have a data field where I could, well, write what is important or not about the family” (D10).
(e)
It was possible to retrace some records of specialists of other institutions in GIS, which included some content about client’s positive changes, his achievements, cooperation results, for example, case management assistance plan. Nevertheless, when we asked survey participants which information about client, in their opinion, was the most important for investigating reports about possible violation of child rights and asked to mark only three answer choices, informants gave priority to entries which contained information about problematic behavior of the clients. In the first place, it was previously recorded possible violations of child rights; in the second place, earlier or actual case management assigned for a family; in the third place, opinion of parents about the circumstances of possible violation of child rights. Meanwhile, case management plan, which covered positive aspects of client’s situation as well, together with several other sources of information took the fifth place.
(f)
Available information that was negatively connotated may have biased the attitude of CRPAS professionals towards the client which could be transmitted further to their decision-making and actions. The traces of such predispositions could be noticed in oral accounts of research participants, for example: “First of all, we take a look at GIS, is it a problematic family or not, what information is there about it and then we go” (D3) or “[In the cases when you find data on IS that a case management was already launched and] there is some work going on with the family, [and the reports come repeatedly], it means that there is something wrong with the family” (D1). In this case, IS was given a power to judge that a client was “normal” or “not normal”, “good” or “bad”. Having in mind that information, which is collected and accessed more simply, more often includes problematic behavior of client, the chances of client to receive an unbiased evaluation are low. To conclude, we have found out that information available in IS supports decision-making of CRPAS professionals and is sufficient for reproducing a detailed picture about the client. However, available information cannot be fully trusted due to occurring errors and imprecisions, and above all due to content of information which describes client as a problematic character. Thus, CW professionals are hardly able to have unbiased attitude towards the client.

4.4. Collecting Activity Evidence by Using the Function of Intermediate Results

Literature review revealed that often the reason behind implementation of IS in different human service organizations is increasing efficiency of accountability rather than improving quality of provided services. Our interview data demonstrated how IS contributes to activation of certain control mechanisms which ensure accountability of CW professionals for their activities.
At CRPAS, work tasks for employees are formulated using IIS. Senior managers formulate the tasks for practitioners of different professions and monitor their performance. Task accomplishment is monitored by using IIS intermediate result function. Practitioners admitted that recording an intermediate result in the system offers senior managers an opportunity to monitor and ensure that decisions and entered data meet the standards and are correct: “because we all are human, and, er, it really happens that in some place you either do not do something, or you forget, […] managers see that and send the task back for accomplishing it to the end. Something hasn’t been done. What means that it is better that we fix it right away, isn’t it, rather than writing an explanation afterwards, after some time” (D9). Well-ordered documentation and evidence-based decisions are extremely important in critical situations when the clients send their complaints: “If there are some complaints, our center [asks]: what did you do? Where did you go? When was it done? Then you can defend yourself, that, see, we have done everything that was needed. What was needed and what was not needed” (D3). Recording an intermediate result also proves convenient when the case is repetitive or when the case has to be overtaken by another employee due to various reasons: “It’s important [to see], what work has already been done, what has been done, what worked well, what didn’t, what actions were applied and et cetera, for not to lose the context and not to treat the case like it was a new one” (D17).
Nevertheless, this function also has some problematic aspects. Informants reported that sometimes they record intermediate results in order to show that they were occupied, especially when they are not able to accomplish the task due to lack of client’s will to cooperate: “I record in the intermediates also why I haven’t brought down that task yet which I can’t get to the end because the father can’t come. He promised and didn’t come, promised and didn’t come. So, I record this and if somebody wants, he will take a look, what has been done with that task of mine. This means that I have contacted and made calls over there, that I have agreed and [he] did not come, promised and did not come, um, finally has put his signature, and then I put it down, but one can see how many actions I have undertaken” (D1). It should be noted that concern of practitioners to record evidence about actions in regard to their clients may be related to tight control at work.
We asked the respondents to rate the level of control at work from 0 to 10, where 0 meant absolute absence of employee control, 5 meant partial control where actions of employee are trusted in general, and 10 meant over control. Distribution of responses was as follows: average rating by professionals, who react to calls and investigate possible violations of child rights, was 7.235, median stood at 8 and IQR was (5, 9); respective indicators of responses by professionals who treat legal, custody and similar questions were 6.62, 6 and (5, 8) and those of employees in manager positions, 6.85, 7 and (5, 8). It should be noted that frontline practitioners engaged in direct work with clients indicated stricter control therefore this group of employees may be inclined to use the function of intermediate results more often than other employees in order to prove their activity.
In summary, data of empirical research showed that, first, function of recording intermediate result allows the managers monitoring work of practitioners with clients and ensure the quality of decisions and of documented information as well as that they meet the standards, what is crucial in critical situations or conflicts with clients. Secondly, this function permits tracking activities of a professional and ensure the continuity of work with client. Thirdly, due to rather strict control at work, practitioners whose work functions are mostly orientated to direct work with client, treat recording of intermediate result as a way to prove their intense activity to managers.

5. Discussion

Systematic literature review revealed that problems related to the effect of IS on interactions of CW professionals with their clients, more precisely their possibilities to allocate enough of time for direct work with the client, were discussed rather widely [15,21,43,46]. For example, Huuskonen and Vakkari [7] stated that social professionals who work in CW field spend more time for producing information than for using it [7]. Our empirical findings confirmed the insights of Shaw et al. [12], Huuskonen and Vakkari [49] about data duplication phenomenon in data collection and entering phases which increases time consumption. We also circumscribed more detailed empirical consequences of multiplication of data collection tools, procedures and storages on working time resources and CW professionals’ possibility to give their time for direct work with clients. Empirical data about technical IS problems and their effect on possibility of CW professionals to serve needs of client and not those of IS have confirmed that time is wasted not only due to human errors or imprecisions in IS use, but also due to inherent IS problems [12]. Despite the fact that some studies (for example, [15]) sought to merge perspectives of user experience and system usability [50], systematic literature review demonstrated that these studies tended to provide sociological explanation of interactions between IS and its user (on micro level) and between IS and the whole organization (on macro level). Our aim in this article is to show the role of IS as a technological construct with certain reactive material properties in the above-mentioned interactions. We contribute to scientific discussion about effect of IS on activity of their users, working environment, possibility to set priorities, goals and tasks [51], professional knowledge of social work practitioners [52], and also on decision-making of CW professionals [11] as well as their discourse about client [44] by providing evidence-based insights about effect of IS on the bias of attitudes of professionals working in CW field towards their client. According to Aiken [53] the professionals’ attitudes in clinical social work may be viewed as learned cognitive, affective, and behavioral predispositions to respond positively or negatively to certain objects, situations, institutions, concepts or persons and influence planning and decision-making. According to empirical research data, information stored in IS is fragmented and reflects only certain and, above all, problematic characteristics of clients. This fact may lead to biased attitudes of CW professionals which, in their turn, may affect decision-making. Our evidence about impact of IS on formation of CW professionals’ attitudes contributes to knowledge base about their effect on decision-making and case outputs under their management [54,55].
Our empirical research data confirmed that higher-level managers use IS for controlling and monitoring the performance of frontline practitioners, in some cases in order to ensure that they meet established standards [43] or to request practitioners’ responsibility for their actions in critical cases [56]. Empirical research data about IS application for guaranteeing continuity of work with client when the case repeats itself or when the case is overtaken by another practitioner contributes to the research field which analyzes IS from temporal perspective [57] by disclosing evolvement of a case over time. Results of empirical research allow to dig deeper into the discussion about IS application in micro-management of professional practice which presupposes increasing management control of frontline practice [58]. Data analysis revealed that overcontrol of frontline practitioners puts a pressure on them to prove their activity by recording intermediate activity results in IS. In this way, at micro level interaction (practitioner—manager), the object of accountability is not performance or quality of provided services or indicators of client welfare anymore, but the evidence that employee was occupied and that “something” has been done. Thus, a bureaucratic model of accountability is established when accountability takes place solely in order to meet the requirements of managers [45]. Future research could make a contribution to more detailed exploration of risks and consequences of recording intermediate results of work with client which does not represent real achievements of work but actions for the sake of formalized accountability.

6. Conclusions

Our empirical research results revealed that the IS used in Lithuanian Child Rights Protection and Adoption Service does not meet certain requirements. We identified the following failures of objective functional requirements: (a) lack of mobile application with a connection to IS; (b) lack of automated error and potential anomalies’ monitoring systems. The non-functional requirements that do not meet the subjective expectations of employees were the following: (c) slow operation and functionality failures due to surplus number of connected users; (d) restrictive search parameters and ambiguous search results; (e) user interface maladapted to data connection.
Different IS requirements which raise challenge to the work practice of child welfare professionals aggravate data duplication problem on three levels: (a) repetition of some information part in different data collection paper forms; (b) repetitive procedures of data collection (a practitioner repeats and enters information about the client several times); (c) a practitioner puts the information in several dissociated/unconnected data storages. Several other drawbacks, directly unrelated to data duplication, such as slow IS operation due to surplus number of users, impractical functionality of data search, inexact information and shortage of other functions, increases dissatisfaction of practitioners and their mistrust in IS, forcing them to search for other informal ways, places and measures to keep the data and reach them without additional difficulties.
In its turn, data duplication (a) increases waste of time (due to forced repetition of data collection and/or input); (b) reduces ecology of work place (due to excess paper consumption); (c) reduces information reliability due to occurring errors; (d) raises additional challenges for data protection.
The problems created by the absence of functionality or its potential failures have their effect on the interaction of CW professionals with their clients and managers. Spending the time for administrative tasks diverts welfare professionals from possibility of direct involvement in the process of helping the client. Functional and non-functional IS requirements which determine the accessibility, completeness, structure and reliability of the information, sometimes raise challenges for practitioners to stay unbiased with their clients and also create a risk to be exposed to overcontrol of managers. Consequently, this leads to formalized accountability instead of ensuring the quality of provided services.

Author Contributions

Conceptualization, M.S.; Data curation, M.B. and S.V.; Formal analysis, M.S. and L.P.; Methodology, M.S. and L.P.; Supervision, D.P.; Visualization, M.B. and S.V.; Writing—original draft, M.S., D.P., M.B., S.V. and L.P.; Writing—review & editing, D.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Vilnius University researchers’ science promotion grant No. MSF-JM-10/2020.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data sharing not applicable.

Acknowledgments

The authors are greatly thankful to State Child Rights Protection and Adoption Service under the Ministry of Social Security and Labour, for favorable conditions and all necessary support for realizing the research.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. Data sources accepted for analysis.
Table A1. Data sources accepted for analysis.
Source NameQC1QC2QC3QC4QC5Score
[48]111115
[45]111115
[46]111115
[17]111115
[44]111115
[43]111115
[49]111115
[1]111115
[47]111115
[40]111115
[15]111115
[42]111115
[59]111115
[60]111014
[7]011114
[61]011114
[21]110114
[20]110114
[62]111014
[63]110114
[64]110114
[65]110114
[66]110114
[67]011013
[68]011103
[69]011013
[70]111003
[71]111003
[72]111003
[30]011013
[73]001113
[74]101013
[68]010113
[75]011002
[31]010012
[76]110002
[77]100102
[78]110002
[79]10.50001.5
[80]001001
[81]100001
[82]100001
[83]100001
[84]100001
[85]010001
[86]100001
[87]000000
[88]000000

References

  1. Gillingham, P. The development of electronic information systems for the future: Practitioners, ‘embodied structures’ and ’technologies-in-practice‘. Br. J. Soc. Work 2013, 43, 430–445. [Google Scholar] [CrossRef]
  2. Rafferty, J.; Steyaert, J. Social work in the digital age. Br. J. Soc. Work 2009, 58, 589–598. [Google Scholar] [CrossRef] [Green Version]
  3. Savaya, R. The potential and utilization of an integrated information system at a family and marriage counselling agency in Israel. Eval. Program Plan. 1998, 21, 11–20. [Google Scholar] [CrossRef]
  4. Dobmeyer, T.W.; Woodward, B.; Olson, L. Factors supporting the development and utilization of an outcome-based performance measurement system in a chemical health case management program. Adm. Soc. Work 2002, 26, 25–44. [Google Scholar] [CrossRef]
  5. Lyons, P.; Doueck, H.J.; Koster, A.J.; Witzky, M.K.; Kelly, P.L. The child well-being scales as a clinical tool and a management information system. Child Welfare 1999, 78, 241–258. [Google Scholar]
  6. Gillingham, P. From bureaucracy to technocracy in a social welfare agency: A cautionary tale. Asia Pac. J. Soc. Work Dev. 2019, 29, 108–119. [Google Scholar] [CrossRef]
  7. Huuskonen, S.; Vakkari, P. Client information system as an everyday information tool in child protection work. In Proceedings of the Third Symposium on Information Interaction in Context, New Brunswick, NJ, USA, 18–21 August 2010; pp. 3–12. [Google Scholar]
  8. Moses, T.; Weaver, D.; Furman, W.; Lindsey, D. Computerization and Job Attitudes in Child Welfare. Adm. Soc. Work. 2003, 27, 47–67. [Google Scholar] [CrossRef]
  9. Munro, E. What tools do we need to improve identification of child abuse? Child Abus. Rev. J. Br. Assoc. Study Prev. Child Abus. Negl. 2005, 14, 374–388. [Google Scholar] [CrossRef] [Green Version]
  10. Peckover, S.; Hall, C.; White, S. From policy to practice: The implementation and negotiation of technologies in everyday child welfare. Child. Soc. 2009, 23, 136–148. [Google Scholar] [CrossRef]
  11. White, S.; Broadhurst, K.; Wastell, D.; Peckover, S.; Hall, C.; Pithouse, A. Whither practice-near research in the modernization programme? Policy blunders in children’s services. J. Soc. Work Pract. 2009, 23, 401–411. [Google Scholar] [CrossRef]
  12. Shaw, I.; Bell, M.; Sinclair, I.; Sloper, P.; Mitchell, W.; Dyson, P.; Clayden, J.; Rafferty, J. An exemplary scheme? An evaluation of the Integrated Children’s System. Br. J. Soc. Work 2009, 39, 613–626. [Google Scholar] [CrossRef] [Green Version]
  13. Ince, D.; Griffiths, A. A chronicling system for children’s social work: Learning from the ICS failure. Br. J. Soc. Work 2011, 41, 1497–1513. [Google Scholar] [CrossRef]
  14. Leonardi, P.M.; Barley, S.R. Materiality and change: Challenges to building better theory about technology and organizing. Inf. Organ. 2008, 18, 159–176. [Google Scholar] [CrossRef]
  15. Huuskonen, S.; Vakkari, P. I Did It My Way: Social workers as secondary designers of a client information system. Inf. Process. Manag. 2013, 49, 380–391. [Google Scholar] [CrossRef]
  16. Savaya, R.; Monnickendam, M.; Waysman, M. Extent and type of worker utilization of an integrated information system in a human services agency. Eval. Program Plan. 2006, 29, 209–216. [Google Scholar] [CrossRef]
  17. Boothby, N.; Stark, L. Data surveillance in child protection systems development: An Indonesian case study. Child Abus. Negl. 2011, 35, 993–1001. [Google Scholar] [CrossRef]
  18. Burton, J.; Van den Broek, D. Accountable and countable: Information management systems and the bureaucratization of social work. Br. J. Soc. Work 2009, 39, 1326–1342. [Google Scholar] [CrossRef]
  19. Slavin, S.; Schoech, R. Human Services Technology: Understanding, Designing, and Implementing Computer and Internet Applications in the Social Services; CRC Press: Boca Raton, FL, USA, 1999. [Google Scholar]
  20. Fitch, D. An Organizational Cybernetics Framework for Achieving Balance in Evidence-Based Practice and Practice-Based Evidence. J. Evid. Based Soc. Work 2014, 11, 114–126. [Google Scholar] [CrossRef]
  21. Wastell, D. Managing as designing: Opportunity knocks for the IS field. Eur. J. Inf. Syst. 2010, 19, 422–431. [Google Scholar] [CrossRef]
  22. Whitworth, B. The social requirements of technical systems. In Virtual Communities: Concepts, Methodologies, Tools and Applications; IGI Global: Hershey, PA, USA, 2011; pp. 1461–1481. [Google Scholar]
  23. Piccoli, G.; Pigni, F. Information Systems for Managers: Without Cases; Prospect Press: Burlington, VT, USA, 2019. [Google Scholar]
  24. Paech, B.; Kerkow, D. Non-functional requirements engineering-quality is essential. In Proceedings of the 10th International Workshop on Requirments Engineering Foundation for Software Quality, Riga, Latvia, 7–8 June 2004. [Google Scholar]
  25. Meijer, P.C.; Verloop, N.; Beijaard, D. Multi-method triangulation in a qualitative study on teachers’ practical knowledge: An attempt to increase internal validity. Qual. Quant. 2002, 36, 145–167. [Google Scholar] [CrossRef]
  26. Kaplan, B.; Duchon, D. Combining qualitative and quantitative methods in information systems research: A case study. MIS Q. 1988, 12, 571–586. [Google Scholar] [CrossRef]
  27. Nightingale, A.J. Triangulation. In International Encyclopedia of Human Geography; Elsevier: Amsterdam, The Netherlands, 2020; pp. 477–480. [Google Scholar]
  28. Aaltrichter, H.; Feldman, A.; Posch, P.; Somekh, B. Teachers Investigate Their Work: An Introduction to Action Research Across the Professions; Routledge: London, UK, 2008. [Google Scholar]
  29. Alassafi, M.O.; Alharthi, A.; Walters, R.J.; Wills, G.B. A framework for critical security factors that influence the decision of cloud adoption by Saudi government agencies. Telemat. Inform. 2017, 34, 996–1010. [Google Scholar] [CrossRef] [Green Version]
  30. Liedgren, P.; Elvhage, G.; Ehrenberg, A.; Kullberg, C. The use of decision support systems in social work: A scoping study literature review. J. Evid. Inf. Soc. Work 2016, 13, 1–20. [Google Scholar] [CrossRef] [PubMed]
  31. Saxena, D.; Badillo-Urquiola, K.; Wisniewski, P.J.; Guha, S. A Human-Centered Review of Algorithms used within the US Child Welfare System. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25–30 April 2020; pp. 1–15. [Google Scholar]
  32. Health, N.; Council, M.R. How to Review the Evidence: Systematic Identification and Review of the Scientific Literature; National Health & Medical Research Council: Canberra, Australia, 2000. [Google Scholar]
  33. Kitchenham, B.; Charters, S. Guidelines for Performing Systematic Literature Reviews in Software Engineering; Technical Report 1; Keele University: Newcastle, UK; University of Durham: Durham, UK, 2007. [Google Scholar] [CrossRef]
  34. Gillingham, P. Computer-based information systems and human service organisations: Emerging problems and future possibilities. Aust. Soc. Work 2011, 64, 299–312. [Google Scholar] [CrossRef]
  35. Latour, B. Reassembling the Social: An Introduction to Actor-Network-Theory; Oxford University Press: Oxford, UK, 2005. [Google Scholar]
  36. Berg, B.L.; Lune, H.; Lune, H. Qualitative Research Methods for the Social Sciences; Pearson: Boston, MA, USA, 2004; Volume 5. [Google Scholar]
  37. Kling, R. What is social informatics and why does it matter? Inf. Soc. 2007, 23, 205–220. [Google Scholar] [CrossRef]
  38. Vehovar, V. Social informatics: An emerging discipline? In Proceedings of the IFIP International Conference on Human Choice and Computers, Maribor, Slovenia, 21–23 September 2006; Springer: Boston, MA, USA, 2006; pp. 73–85. [Google Scholar]
  39. Mayring, P. Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution; Browse Publications: Klagenfurt, Austria, 2014. [Google Scholar]
  40. Huuskonen, S.; Vakkari, P. Client’s temporal trajectory in child protection: Piecing information together in a client information system. In Proceedings of the IFIP Conference on Human-Computer Interaction, Lisbon, Portugal, 5–9 September 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 152–169. [Google Scholar]
  41. Gillingham, P. Information Systems and Human Service Organizations: Managing and Designing for the “Occasional User”. Hum. Serv. Organ. Manag. Leadersh. Gov. 2014, 38, 169–177. [Google Scholar] [CrossRef]
  42. Lagsten, J.; Andersson, A. Use of information systems in social work-challenges and an agenda for future research. Eur. J. Soc. Work 2018, 21, 850–862. [Google Scholar] [CrossRef]
  43. Gillingham, P. Electronic Information Systems and Human Service Organizations: The Unanticipated Consequences of Organizational Change. Hum. Serv. Organ. Manag. Leadersh. Gov. 2015, 39, 89–100. [Google Scholar] [CrossRef]
  44. Gillingham, P. Electronic Information Systems in Human Service Organisations: The What, Who, Why and How of Information. Br. J. Soc. Work 2015, 45, 1598–1613. [Google Scholar] [CrossRef]
  45. Bach-Mortensen, A.M.; Montgomery, P. What are the barriers and facilitators for third sector organisations (non-profits) to evaluate their services? A systematic review. Syst. Rev. 2018, 7, 13. [Google Scholar] [CrossRef] [Green Version]
  46. Naqvi, D.; Malik, A.; Al-Zubaidy, M.; Naqvi, F.; Tahir, A.; Tarfiee, A.; Vara, S.; Meyer, E. The general practice perspective on barriers to integration between primary and social care: A London, United Kingdom-based qualitative interview study. BMJ Open 2019, 9. [Google Scholar] [CrossRef]
  47. Lecluijze, I.; Penders, B.; Feron, F.J.; Horstman, K. Co-production of ICT and children at risk: The introduction of the Child Index in Dutch child welfare. Child. Youth Serv. Rev. 2015, 56, 161–168. [Google Scholar] [CrossRef]
  48. Godara, B.; Kumar, N.; Boursin, F.; Jobit, G.; Amara, A.; Agagliate, T. ICTs as catalysts in child protection programmes: Current landscape in South Asia & a concept to inform future use. In Proceedings of the Tenth International Conference on Information and Communication Technologies and Development, Ahmedabad, India, 4–7 January 2019; pp. 1–6. [Google Scholar]
  49. Huuskonen, S.; Vakkari, P. Selective clients’ trajectories in case files: Filtering out information in the recording process in child protection. Br. J. Soc. Work 2015, 45, 792–808. [Google Scholar] [CrossRef]
  50. Bevan, N. Classifying and selecting UX and usability measures. International Workshop on Meaningful Measures: Valid Useful User Experience Measurement, Reykjavik, Iceland, 18 June 2008; Volume 11, pp. 13–18. [Google Scholar]
  51. Ley, T.; Seelmeyer, U. Professionalism and information technology: Positioning and mediation. Soc. Work Soc. 2008, 6, 338–351. [Google Scholar]
  52. Parton, N. Changes in the form of knowledge in social work: From the ‘social’to the ‘informational’? Br. J. Soc. Work 2008, 38, 253–269. [Google Scholar] [CrossRef]
  53. Aiken, L. Attitudes and Related Psychosocial Constructs: Theories, Assessment, and Research; Sage: Newbury Park, CA, USA, 2002. [Google Scholar]
  54. Benbenishty, R.; Davidson-Arad, B.; López, M.; Devaney, J.; Spratt, T.; Koopmans, C.; Knorth, E.J.; Witteman, C.L.; Del Valle, J.F.; Hayes, D. Decision making in child protection: An international comparative study on maltreatment substantiation, risk assessment and interventions recommendations, and the role of professionals’ child welfare attitudes. Child Abus. Negl. 2015, 49, 63–75. [Google Scholar] [CrossRef] [Green Version]
  55. Arroyo, J.; Peek, C.W. Child welfare caseworkers’ characteristics and their attitudes toward non-custodial fathers. Child Abus. Negl. 2015, 47, 140–152. [Google Scholar] [CrossRef]
  56. Munro, E. The Munro Review of Child Protection: Final Report, a Child-Centred System; The Stationery Office: London, UK, 2011; Volume 8062. [Google Scholar]
  57. Reddy, M.C.; Dourish, P.; Pratt, W. Temporality in medical work: Time also matters. Comput. Supported Coop. Work 2006, 15, 29–53. [Google Scholar] [CrossRef]
  58. Wastell, D.; White, S. Beyond bureaucracy: Emerging trends in social care informatics. Health Inform. J. 2014, 20, 213–219. [Google Scholar] [CrossRef]
  59. Nielsen, M.B.; Christensen, J.O.; Hetland, J.; Finne, L.B. Organizational Prevention and Management Strategies for Workplace Aggression Among Child Protection Workers: A Project Protocol for the Oslo Workplace Aggression Survey (OWAS). Front. Psychol. 2020, 11. [Google Scholar] [CrossRef]
  60. Harrison, T.M.; Canestraro, D.; Pardo, T.; Avila-Marilla, M.; Soto, N.; Sutherland, M.; Burke, B.; Gasco, M. A tale of two information systems: Transitioning to a data-centric, information system for child welfare. In Proceedings of the 19th Annual International Conference on Digital Government Research: Governance in the Data Age, Delft, The Netherlands, 30 May–1 June 2018. [Google Scholar] [CrossRef]
  61. Harrison, T.; Canestraro, D.; Pardo, T.; Avila-Maravilla, M.; Soto, N.; Sutherland, M.; Burke, G.B.; Gasco, M. Applying an Enterprise Data Model in Government. In Proceedings of the 20th Annual International Conference on Digital Government Research, Dubai, United Arab Emirates, 18–20 June 2019; pp. 265–271. [Google Scholar]
  62. Jang, K. Technology Could Be Harmful Rather Than Beneficial: An Empirical Investigation of Caseworkers’ Perceptions Using a Knowledge Management Framework. J. Soc. Serv. Res. 2015, 41, 246–268. [Google Scholar] [CrossRef]
  63. Gillingham, P. Technology Configuring the User: Implications for the Redesign of Electronic Information Systems in Social Work. Br. J. Soc. Work 2016, 46, 323–338. [Google Scholar] [CrossRef]
  64. Spensberger, F. The digitalization of a social work theory: Experiences of a German child welfare social worker. Int. Soc. Work 2019, 62, 1575–1579. [Google Scholar] [CrossRef]
  65. Gillingham, P. Developments in Electronic Information Systems in Social Welfare Agencies: From Simple to Complex. Br. J. Soc. Work 2019, 49, 135–146. [Google Scholar] [CrossRef]
  66. De Corte, J.; Devlieghere, J.; Roets, G.; Roose, R. Top-down policy implementation and social workers as institutional entrepreneurs: The case of an electronic information system in Belgium. Br. J. Soc. Work 2019, 49, 1317–1332. [Google Scholar] [CrossRef]
  67. Bopp, C.; Benjamin, L.M.; Voida, A. The coerciveness of the primary key: Infrastructure problems in human services work. Proc. ACM Hum. Comput. Interact. 2019, 3, 1–26. [Google Scholar] [CrossRef] [Green Version]
  68. Vogl, T.M. Artificial intelligence and organizational memory in government: The experience of record duplication in the child welfare sector in Canada. In Proceedings of the 21st Annual International Conference on Digital Government Research, Seoul, Korea, 15–19 June 2020; pp. 223–231. [Google Scholar]
  69. Rolan, G.; Phan, H.D.; Evans, J. Recordkeeping and Relationships: Designing for Lifelong Information Rights. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, Eindhoven, The Netherlands, 6–10 July 2020; pp. 205–218. [Google Scholar]
  70. Cronley, C.; Patterson, D.A. How well does it fit? an organizational culture approach to assessing technology use among homeless service providers. Adm. Soc. Work 2010, 34, 286–303. [Google Scholar] [CrossRef]
  71. Savaya, R.; Altschuler, D.; Melamed, S. Apprehensions and Expectations of the Adoption of Systematically Planned, Outcome-Oriented Practice. Res. Soc. Work Pract. 2013, 23, 437–446. [Google Scholar] [CrossRef]
  72. Patterson, D.A.; Cronley, C.; West, S.; Lantz, J. Social justice manifest: A university-community partnership to promote the individual right to housing. J. Soc. Work Educ. 2014, 50, 234–246. [Google Scholar] [CrossRef]
  73. Bilson, A.; Cant, R.L.; Harries, M.; Thorpe, D.H. Accounting for the increase of children in care in western Australia: What can a client information system tell us? Child Abus. Negl. 2017, 72, 291–300. [Google Scholar] [CrossRef]
  74. Wang, C.; Mody, E.H.; Hunting, D.; Hoyt, J.; Ferguson, K.M. Data-driven outreach to opportunity youth using population data and Geographic Information System technology. J. Soc. Work 2020. [Google Scholar] [CrossRef]
  75. Kotoulas, S.; Lopez, V.; Sbodio, M.L.; Stephenson, M.; Tommasi, P.; Mac Aonghusa, P. A linked data approach to care coordination. In Proceedings of the 25th ACM Conference on Hypertext and Social Media, Santiago, Chile, 1 September 2014; Association for Computing Machinery: New York, NY, USA, 2014; pp. 77–87. [Google Scholar] [CrossRef]
  76. Devlieghere, J.; Roose, R. Documenting practices in human service organisations through information systems: When the quest for visibility ends in darkness. Soc. Incl. 2019, 7, 207–217. [Google Scholar] [CrossRef] [Green Version]
  77. Munro, E. Learning to reduce risk in child protection. Br. J. Soc. Work 2010, 40, 1135–1151. [Google Scholar] [CrossRef] [Green Version]
  78. Cash, S.J.; Ingram, S.D.; Biben, D.S.; McKeever, S.J.; Thompson, R.W.; Ferrell, J.Z. Moving forward without looking back: Performance management systems as real-time evidence-based practice tools. Child. Youth Serv. Rev. 2012, 34, 655–659. [Google Scholar] [CrossRef]
  79. Jucan, C.N.; Jucan, M.S. Remodeling artificial intelligence to ethics and social responsibility. Leg. Pract. Int. Laws 2011, 179–186. [Google Scholar]
  80. Zimic, S.; Dalin, R. Systematical follow-up in social work practices. ACM Sigcas Comput. Soc. 2016, 45, 159–166. [Google Scholar] [CrossRef]
  81. Woodman, J.; Gilbert, R. Proposed Child Protection Information System seems to run counter to best evidence. BMJ 2013. [Google Scholar] [CrossRef] [Green Version]
  82. Lyons, P.; Winter, C.L. Data management system selection in a family service agency. Fam. Soc. 2010, 91, 440–446. [Google Scholar] [CrossRef]
  83. Frerer, K.; Sosenko, L. Overcoming barriers to education and child welfare data sharing: Lessons learned from the California ready to succeed pilot project. In Proceedings of the 5th International Technology, Education and Development Conference, Valencia, Spain, 7–9 March 2011; pp. 6604–6610. [Google Scholar]
  84. Dysart-Gale, D.; Pitula, K.; Radhakrishnan, T. Culture, communication, and ICT for development: A caribbean study. IEEE Trans. Prof. Commun. 2011, 54, 43–55. [Google Scholar] [CrossRef]
  85. Marlow, J. Impression formation in social work-sharing sites. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, San Antonio, TX, USA, 23–27 February 2013; pp. 69–72. [Google Scholar] [CrossRef]
  86. Hall, J.A.; Imburgia, T.M.; Bloomquist, K.R.; Kim, J.; Pierce, B.J.; Jaggers, J.W.; Armstrong-Richardson, E.; Danh, M.; Hensel, D.J. Partnership for Multimethod Evaluation in Child Welfare: Title IV-E Waiver Demonstration Program. Child Welf. 2017, 95, 59–78. [Google Scholar]
  87. Wolff, T.B. The USC school of social work faculty technology workshop. In Proceedings of the SIGUCCS’12—ACM Proceedings of the SIGUCCS Annual Conference, Memphis, TN, USA, 15–19 October 2012; pp. 165–167. [Google Scholar] [CrossRef]
  88. Duarte, A.I.M.; Costa, C.J. Information Systems: Life Cycle and Success; ACM International Conference Proceeding Series; ACM: New York, NY, USA, 2012; pp. 25–30. [Google Scholar] [CrossRef]
Figure 1. Flowchart of of the article selection process.
Figure 1. Flowchart of of the article selection process.
Sustainability 13 06765 g001
Figure 2. Frequency of Technical Issues of IS (1-never, 6-always).
Figure 2. Frequency of Technical Issues of IS (1-never, 6-always).
Sustainability 13 06765 g002
Figure 3. Level of Detail of Client’s Picture Reproduced According to Information in IS.
Figure 3. Level of Detail of Client’s Picture Reproduced According to Information in IS.
Sustainability 13 06765 g003
Table 1. Inclusion and Exclusion Criteria for Systematic Literature Review.
Table 1. Inclusion and Exclusion Criteria for Systematic Literature Review.
NameInclusion CriteriaExclusion Criteria
YearFrom 2010 (inclusive)Until 2010
LanguageEnglish and LithuanianOther languages
ContentChild welfare and ISOther topics
Table 2. Search Strings for Systematic Literature Review.
Table 2. Search Strings for Systematic Literature Review.
Search Strings
QR1“information system” AND “social work”
QR2“information system” AND “child protection”
QR3“information system” AND “child welfare”
QR4“management system” AND “child protection”
QR5“management system” AND “child welfare”
Table 3. Actual and Desired Allocation of CRPAS Professionals’ Working Time.
Table 3. Actual and Desired Allocation of CRPAS Professionals’ Working Time.
Actual AllocationDesired AllocationDifference (Percentage Points)
Computer work (filling questionnaires and forms, entering data into IS, document management and communication by e-mails, report preparation)44.3%30.0%+14.3
Direct work with clients (listening to opinions of parents and child, family visits, client consultations, participating in case management meetings etc.)39.3%48.7%−9.4
Team work (formal/informal group reflexions, case discussions, intervisions, organizational meetings)13.4%18.2%−4.8
Other tasks3.0 %2.9 %+0.1
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Seniutis, M.; Petružytė, D.; Baltrūnaitė, M.; Vainauskaitė, S.; Petkevičius, L. The Impact of Information System on Interactions of Child Welfare Professionals with Managers and Clients. Sustainability 2021, 13, 6765. https://doi.org/10.3390/su13126765

AMA Style

Seniutis M, Petružytė D, Baltrūnaitė M, Vainauskaitė S, Petkevičius L. The Impact of Information System on Interactions of Child Welfare Professionals with Managers and Clients. Sustainability. 2021; 13(12):6765. https://doi.org/10.3390/su13126765

Chicago/Turabian Style

Seniutis, Miroslavas, Donata Petružytė, Miglė Baltrūnaitė, Silvija Vainauskaitė, and Linas Petkevičius. 2021. "The Impact of Information System on Interactions of Child Welfare Professionals with Managers and Clients" Sustainability 13, no. 12: 6765. https://doi.org/10.3390/su13126765

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop