Library Assessment Research: A Content Comparison from Three American Library Journals

Abstract

Improvement of academic library services as an outcome of continuous assessment is an aim of libraries of higher education institutions. Academic libraries are realizing the need to document evidence of their value to the institutions and the patrons they serve. Publications that include assessment research are reaching library decision makers, who seek to apply evidence to improve services or implement best practices that benefit all stakeholders. Following two previous studies that reported longitudinally on front-line library services, this paper investigates current five-year trending of three prestigious academic library journals in the publication of assessment studies. Data for this study were drawn through a content analysis process, in which the investigators selected studies for inclusion using a set of criteria developed in a pilot exercise. After individually examining 649 research articles, published between 2012 and 2016, 126 met the study’s selection criteria and were categorized according to the type of service they studied. Papers on information literacy instruction dominated, while reference services, technology, and general assessment studies saw less representation in the three journals. This finding reflects the priority placed upon information literacy instruction and describes how three American library journals are responding to current trends across academic libraries.

1. Introduction

Academic libraries within institutions of higher education can offer an impressive array of front-line services, inclusive of reference, instruction, and computer technology, but they are also moving toward futuristic, leading-edge services and offerings, such as big data curation, digitalization, makerspaces, and scholarly communications services, to name a few. Not all academic libraries, however, offer these services uniformly, and considering whether an institution grants degrees at the doctoral, bachelor’s, or associate’s level, the nature and quantity of services themselves may vary in response to differentiated needs. How library patrons use the services and resources of their home institution, and patrons’ satisfaction with them, have been a documented concern for library decision-makers. Service and resource assessments provide library decision-makers evidence as they strive to make improvements, and when assessment studies are published, the knowledge base for them is expanded. Hernon and Dugan describe the rationale for studying service quality and patron satisfaction:

Either service quality or satisfaction can be an end in itself; each is worthy of examination as a framework for evaluating library services from a customer’s or user’s perspective. By paying proper attention to assessment, service quality, and satisfaction, libraries are, in effect, promoting continuous quality improvement. Improvements lead to change, and library leaders must manage that change and ensure that the library’s assessment plan is realistic and realized. Both service quality and satisfaction should be part of any culture of assessment and evaluation.

[1] (p. 120)

The improvement of services, particularly in the front-line areas of information literacy instruction, reference, and technology, have become the focus of both informal self-study and published assessments. Studies of this type may be found in journals devoted to specialized services, but also in journals of high impact and wide readership. Examples of the latter are Journal of the Association for Information Science and Technology, Journal of Documentation, and New Review of Academic Librarianship. Examples of specialized journals not limited to an academic library readership that cover assessments are the Journal of Interlibrary Loan, Document Delivery, & Electronic Reserve and Cataloging & Classification Quarterly. There are many more examples of each type of journal dedicated to the promotion of best practices through assessments. Further, there are professional gatherings, such as the International Conference on Performance Measurement in Libraries (formerly called the Northumbria Conference), the Association of Research Libraries (ARL) Library Assessment Conference (LAC), and the Evidence Based Library and Information Practice Conference, which sponsor journals or publish proceedings. These developments are indicative of the desire among library professionals to grow and improve services offered to their patrons.

The relationship between scholars and custodians of texts with respect to service has changed significantly over time. As recently as the 20th century, service to scholars was very limited. In the current academic environment, however, library patrons are valued as “customers”, and consequently their satisfaction with library services takes high priority. The supporting pillar for library customer satisfaction is a culture of assessment, which takes into consideration how well library services are perceived by patrons. Further, assessments are a vehicle for communicating value to the parent institution’s administrators, particularly through the measurement of learning outcomes following library instruction and information literacy programming. The professional literature covers an array of articles on library services, but only within the last four decades has evidence-based research on front-line services been more abundant and available to decision-makers. The problem, however, is that assessment research on front-line academic library services in the last decade reflects a top-heavy presence of studies focusing upon information literacy and learning outcomes. While this content is important, and in fact vital to demonstrating value, studies on other service areas have relatively less representation.

The present study was undertaken to examine the availability of published assessments to college and university library decision-makers as they draw from a data-rich knowledge base. By examining a recent sample of the professional library literature, it aims to identify what may be needed to expand the literature, while providing a snapshot of what is trending within published assessments on library services. The study examines both the coverage of site-specific general library assessments and assessments of three front-line services that have been covered within three prominent Library and Information Science (LIS) journals over a recent five-year period.

2. Literature Review

2.1. Value

The value of academic libraries may be defined in different ways. In Value of Academic Libraries (VAL), Oakleaf summarizes relevant definitions in terms of internal and external focuses. The internal definitions center upon use, return-on-investment, and the production of a commodity, whereas the external focus extends to a more experiential value or impact upon the user—that is, on those activities that the “library enables them to do” [2] (p. 23). Lakos and Phipps, with Wilson, describe the assessment culture as “an organizational environment in which decisions are based on facts, research, analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders” [3] (p. 352).

Value is not only to be experienced by day-to-day student and faculty populations, but also to be communicated to its institutional leaders, as they perceive the library as part of a partnership to realize its goals, outcomes and missions. [4]. Coinciding with the need to advance the value of libraries through more complex assessment methodologies and communication skills, the number of library positions requiring assessment skills has increased [5], with new proficiency standards being published for assessment librarians [6]. With the growth of the assessment movement, a burgeoning of professional development opportunities has taken hold through international conferences, specialized training, and expansion within professional associations to develop this area of librarianship [7], the goals of which are service improvement and an outward demonstration of value.

2.2. Developing a Service Orientation

Assessment policies and procedures vary widely within higher education disciplines and programs. The library, working as an integral partner to fulfill the larger institutional mission, participates by supporting faculty, students, and its community, with numerous services specific to the needs of all its stakeholders. Libraries are also included in the parent institution’s accreditation evaluation process for program licensure and certification requirements. According to Town [8], measurement and evaluation of libraries as a means of demonstrating value to parent institutions and stakeholders has developed significantly over the last one hundred years. Citing Thompson [9], the historical development can be segmented into three phases: storehouse, service, and education. Storehouse refers to evaluation in terms of inputs. That is, quantified data describing what a library has, such as the size of the collection, number of books acquired, salaries paid, expenditures, and the like, such as the ARL Index, or its early predecessor, the Gerould Statistics. It may also refer to evaluation in terms of standards, such as 1928 College Library Standards, or the 1959 ARL Standards [10].

The momentum for the service phase might be traced to F.W. Lancaster’s works on quality improvement and his reliance upon Ranganathan’s Five Laws of Library Science [11]. Lancaster’s background in the area of systems analysis, combined with Duane Webster’s human resources experience at ARL, are described by Kyrillidou and Cook: “contributed to an increased awareness of libraries as symbolic entities manifesting elements of effect of service, information control, and library as place that generate perceptions and expectations as library users come into contact with these entities” [11] (p. 290). Within this same phase, and particularly in the 1980s and 1990s, Total Quality Management (TQM) gained acceptance among academic library managers as a strategic effort to effect improvements. TQM principles were developed for business management, and transferred to libraries, seeking to improve customer satisfaction and quality improvement [12]. Service quality and patron satisfaction, however, were not always an important concern for library managers. McElderry described, from an historical perspective, how library service in American higher education institutions developed through the 19th and early 20th centuries [13]. Essentially, academic libraries were operating on something of a self-service basis, in which practically no services were provided to scholars. It was only when there was a greater demand for higher education that the number of colleges and universities grew, and in order to support expanding curricula and growing subject specializations, larger library collections were needed, and with them, reference services to guide readers through library catalogues. At first, reference service models were slow to develop, but over time, philosophies of service emerged, ranging from the notion that librarians should facilitate self-sufficiency to models that operationalized full delivery on requests for reliable information. By the 1930s and 1940s, service as we understand it today was in its infancy, and “the products and services needed to satisfy reader requirements were not well understood” [13] (p. 418). Academic libraries of the 21st century have very rapidly undergone much change, but along with these changes, has come greater awareness of the need to improve services.

2.3. Measuring Quality

As in the application of TQM business principles to academic library assessments in the 1980s and 1990s, ServQUAL, a quality assessment tool developed for the retail industry, was also adapted to academic libraries. By design, the instrument identifies discrepancies between “the minimum, perceived, and desired levels of performance across five dimensions (tangibles, reliability, responsiveness, assurance, and empathy)” [14] (p. 243). By 1999, ARL’s New Measures Initiative (NMI) was developing a new “toolbox”, or suite of assessment products, that measured not only user satisfaction with library services, but also an evaluation of the efficiency of electronic resource investment. The NMI was renamed the StatsQUAL Gateway, and incorporated both quantitative and qualitative methodologies, through products such as LibQUAL+™, DigiQUAL, MINES for Libraries, and ClimateQUAL. LibQUAL+™ was piloted in 1999 [15], and has been used subsequently as an assessment tool in academic libraries worldwide.

With these tools in hand, many academic libraries have been developing and strengthening cultures of assessment. Studies examining what it takes to sustain a culture of assessment have concluded that library managers play an important role in supporting that culture [16,17], that a customer focus is essential [3,16], and that reliance upon external and evidence-based research should inform decisions [3,18]. Sources for evidence-based research, such as assessment studies, systematic reviews, and evidence summaries, are typically searched in the journal literature. It has also been suggested that a database for evidence summaries be created to assist decision makers and librarianship as a whole [19].

2.4. Assessment Trends

The educational phase, extending to learning metrics and the alignment of assessments with institutional goals, became a significant challenge to academic libraries in institutions of higher education. Interest in assessing information literacy instruction was heightened after the 2005 release of the U.S. Department of Education’s report “A Test of Leadership: Charting the Future of United States Higher Education” [20]. This document affected regional accreditation organizations’ standards, which in turn impacted the focus for assessment within academic and research libraries. The overall effect was to place a greater emphasis upon improvements in student learning in the post-2005 LIS literature. Hufford’s study, published in 2013, reviewed all types of assessment literature between 2005 and 2011. He noted an emphasis on assessments of information literacy instruction, stating that the increase was due to librarians’ belief that this is one of the most important services they provide, and further, predicted that this is a trend that will continue [20] (p. 20).

In 2010, the Association of College and Research Libraries (ACRL) published Value of Academic Libraries: A Comprehensive Research Review and Report (VAL), which covered existing research, research gaps, and the “most promising best practices and measures correlated to performance” for an academic librarian readership [21] (p. 1/1). Oakleaf et al. found, subsequent to the 2010 report, that “In general, the research linking libraries with student learning and success has pursued a correlation approach in which librarians use correlation methodologies to explore connections between library services and resources and the needs, goals, and outcomes of their institutions” [22] (p. 454). The alignment of an assessment research agenda to that of the parent institution’s, as outlined in VAL, encompasses the following areas: student enrollment, student retention and graduation, student success, student achievement, student learning, student experience, faculty research productivity, faculty grants, faculty teaching, and institutional reputation [2] (p. 17). This course of action is not without obstacles, and due to the siloed nature of units and departments in higher educational institutions, there may be challenges to obtaining needed analytic learning data, which would establish correlational evidence of a library’s impact to student success [22]. To date, a resolution to this problem is not widely available.

Earlier evidence of LIS publications carrying instructional research studies comes from the 2007 study by Crawford and Feldt. They discovered that between 1971 and 2002, there were four LIS journals that published a total of more than 50 research articles on library instruction during this time frame. These journals were Research Strategies (RS), Reference Services Review (RSR), Journal of Academic Librarianship (JAL), and College & Research Libraries (CRL). Among them, RS published the highest number of research studies on library instruction, and JAL published the highest number of essays on the same subject [23]. A content analysis by Luo and McKinney in 2015 also identified JAL as a leader in the publication of information literacy, finding that between 2004 and 2013, information literacy was its most popular topic [24].

The 2012 study by Mahraj reported a content analysis of a six-year span (2006–2011) from issues of Reference Services Review (RSR). Although the study did not distinguish between empirical research and other types of articles, it provided a topical breakdown. Information literacy and instruction articles accounted for 49% of the journal’s content. Articles on reference service were 28% of the data set, and emerging technologies saw coverage at 18%. Mahraj states: “While the data does not explain this pattern, the overall volume of content on information literacy is a potential signal of librarians’ shifting roles and priorities within academic institutions and may speak to trending in the profession at large” [25] (p. 185). Clark replicated the Mahraj study in 2015 with a three-year data set consisting of articles from the same journal (2012–2014). The study’s findings showed very similar representation of information literacy and instruction topics in two of the three years under study, but the number of emerging technology articles published between 2012 and 2014 accounted for 30% of articles, with the largest number having been published in 2012. Articles on reference topics dropped to 12% from 28% as compared to the former study. Clark credits the dynamism of the library field and notably changing technologies for shifts taking place in the publication of more diversified topics [26] (pp. 74–75).

3. Methodology

Historically, academic and research libraries have served varied patrons, ranging from university scholars to the incoming undergraduate student body. The perception of value to a library’s stakeholders in the current climate may be enhanced through the delivery of service quality to end-users, or through the demonstration of successful learning outcomes to the administrative leaders of its institution. A solid assessment culture is supported by administrative leadership, but also by the examination of empirical research, whether drawn from periodical literature or through original research. Assessment studies in the areas of overall library satisfaction, information literacy instruction, reference service, and technology within academic libraries are being published in a wide range of specialized and non-specialized LIS journals. However, while it has been noted that information literacy assessments are a means of demonstrating participation in institutional missions and communicating value to administrative stakeholders, published studies on other services to patrons may not share the same level of priority on library research agendas. Recent evidence of the growing movement to demonstrate value through assessment studies is found in the Association of College and Research Libraries’ (ACRL) Assessment in Action (AiA) program and its accompanying RoadShow of traveling workshops [27]. The AiA’s participants in North America and Australia have carried out assessment projects through action research using multiple methodologies. These newer and externally-focused assessment studies are designed to demonstrate value through an alignment with institutional goals and missions. The AiA report, co-authored by Brown and Malenfant, provides the context for the AiA initiative in its Executive Summary. Essentially, the document provides a framework for assessments linking what libraries have to offer (i.e., instruction, reference, collections, space and facilities) to student learning outcomes and overall academic success [28].

The following questions guided the present research:

• To what extent have site-specific, front-line library service and general satisfaction assessments studies been published by three internationally recognized, high-impact academic library journals within a recent 5-year period?
• How does the coverage of criteria-qualifying assessment content compare among the three journals?

3.1. Materials

This study was designed to provide an in-depth investigation into the practices adopted by academic libraries specific to published assessments. The framework for continuous improvement within academic libraries is a culture of assessment, focusing on the collection of evidence for the decisions that ultimately affect numerous stakeholders. Three well-recognized, peer-reviewed American journals covering topics of interest to academic library practitioners were selected for this descriptive content analysis study. The Journal of Academic Librarianship (JAL) has a five-year impact factor of 1.395, portal: Libraries and the Academy (portal) has an impact factor of 1.29 over a four-year period, and College & Research Libraries (CRL) had a 2016 Impact Factor of 1.515. CRL is the only one of the three that is published in open access. These journals represent major publications that may influence the direction of assessment and implementations of change within higher educational institutions. Furthermore, the articles published within these three journals were suitable for meeting the study’s selection criteria. They provide a foundation of policies and procedures that may close the gap between front-line library services and the needed evidence-based documentation to envision improvement of academic library services. Other peer-reviewed LIS journals with similar scopes were initially considered for inclusion in the study, but due to the granular nature of the data collection process and time limitations, they were excluded from the present study.

A literature search exercise, designed to lead staff to finding empirical evidence for reducing reported library barriers, was the starting point for the current study. The term “empirical” here is defined broadly as any type of data collected for the purpose of research. As searching progressed, the topic was narrowed to assessments that produced data that can be used to implement improvements in services to patrons. Initially, the search was conducted by scanning tables of the content of LIS journals of limited aims and scope, specifically those that covered management, interlibrary loan, cataloging and classification, reference, collections, and circulation. The sampling then progressed to LIS journals of wider appeal and reader interests. It was observed that research studies on a broader array of services were being published within the journals of wider scope. A decision was made to investigate the contents of three top peer-reviewed journals of the latter type. Studies considered for inclusion were narrowed to those focused on general patron satisfaction assessments and those that examined information literacy, reference, and usability of technology, to the extent that these services had a direct impact on library patrons. Excluded were studies that assessed other library assets and functions, such as collections, facilities, and operations and processes. Studies that involved multiple sites, reviews of the literature, meta-analyses, essays, or opinion articles were also excluded. The finalized criteria for examination were comprised of the following components: publication dates 2012–2016; empirical studies; satisfaction assessment studies of academic libraries; and studies specific to information literacy instruction, reference, and technologies used in academic libraries.

The three journals selected for investigation are read by a large and varied population. JAL is indexed by 16 abstracting and indexing (A&I) services, and has been in publication since 1974. JAL is an international and refereed journal, and publishes articles that focus on problems and issues germane to college and university libraries. JAL provides a forum for authors to present research findings and, where applicable, those findings’ practical applications and significance. Authors also analyze policies, practices, issues, and trends; speculate about the future of academic librarianship; and present analytical bibliographic essays and philosophical treatises [29].

The portal journal is indexed in over 50 A&I databases. From the journal’s website: “Focusing on important research about the role of academic libraries and librarianship, portal also features commentary on issues in technology and publishing. Written for all those interested in the role of libraries within the academy, portal includes peer-reviewed articles addressing subjects such as library administration, information technology, and information policy” [30]. Portal is published by Johns Hopkins University Press and has had a continuous run since 2001.

CRL is the official publication of the ACRL, and is published online-only on a bi-monthly schedule. It has had a longstanding publication run, having published its first issue in 1939. The journal is indexed by no less than eight indexers, and articles are freely accessible through internet searching. The scope of the journal extends to “all aspects of academic and research librarianship”, but its main focus is original research [31]. The authors believed these three publications were suitable for content analysis and comparison, due to their respective audiences, reputation, and similar yet distinct aims and scopes.

The methodology for this study originates in Krippendorff’s definition of content analysis, “a research technique for making inferences from texts (or other meaningful matter) to the contexts of their use” [32] (p. 16). Precedents for similar content analysis studies of LIS journals are found in Mahraj and Clark [25,26]. Content analysis is an unobtrusive data collection activity, undertaken in the absence of directly observable evidence [32] (p. 39). It is an indirect method of observation and as such, research questions are answered inferentially.

3.2. Procedures

The initial procedure for developing a codebook and the textual materials under investigation have been described above. The investigators worked together, rather than independently, over many sessions, examining and coding research articles published in JAL, portal, and CRL between 2012 and 2016. A total of 323 research articles published in JAL, 130 articles published in portal, and 196 in CRL were examined. Qualifying articles were documented in spreadsheet entries containing journal volume and issue numbers, year of publication, each article’s title, the relevant area of library service, and the country where the research was conducted. Authors’ names were omitted. Where ambiguities arose, differing interpretations of a given text were discussed in light of the inclusion criteria. Throughout the coding process, and particularly in these instances, texts were keyword-searched for critical terms, such as “assess”, “service”, and “improve”, in order to reach a reasoned determination. Differences of opinion were resolved in this manner, rather than by settling disagreements after independent scoring and calculations of inter-rater consensus estimates [32,33,34]. Due to the descriptive character of the study, no correlational statistical measures were undertaken.

4. Findings

This study was undertaken to identify general satisfaction and front-line library service assessments found within three distinguished, American peer-reviewed journals available to college and university library decision-makers, as they draw from an empirical knowledge base. Studies meeting the selection criteria are identified in Appendix A, Appendix B, and Appendix C, and these entries constitute the full dataset. All data entries were taken from articles published between 2012 and 2016. Each study listed in the appendices was judged to be empirical and fell within the study’s description of an assessment study. Topics covered within the data set were general satisfaction assessments (e.g., LibQUAL+), assessments of information literacy instruction, reference services, and technologies used in academic libraries. Of the 323 articles in JAL, 18% (n = 59) of research articles met the coding criteria. In portal, of 130 articles examined, 23 articles (18%) met the criteria, and of 196 articles in CRL, 44 (22%) qualified for inclusion within the data set. (During the data collection procedure, it was discovered that in portal there was a gap of between volume 14, number 1 and volume 15, number 2 in which there were no studies published covering the library services investigated in the present study. This is likely attributable to a transitional period between editors. In the last issue of volume 14 (2014), the editor announced the end of her tenure and named the immediate successor to the position. With volume 15, number 3 (2015), the authors were able to resume identification of criteria-qualifying studies. Had there not been a break in editorial continuity, it is assumed that the study would likely have provided a larger number of studies. The authors also speculate that empirical, service-study manuscripts not accepted by portal through the transitional gap were either held for future publication or published elsewhere.)

Assessment studies in information literacy dominated all three journals at rates of 61% (n = 36 in JAL), 48% (n = 11 in portal), and 52% (n = 23 in CRL). Six studies on reference services appeared in portal (26%), seven studies were identified in JAL (12%), and eight studies (18%) in CRL. Library technology studies had the greatest presence in CRL, with 11 studies (25%). In portal, five technology studies (22%) were identified, and JAL followed with nine studies (15%). General library satisfaction assessments constituted 10% of studies in JAL (n = 6), 8% (n = 5) in CRL, and less than 1% (n = 1) in portal through the same period (see Figure 1, Figure 2 and Figure 3). Five-year composite trending data from the three journal sources (n = 126) for general satisfaction assessments, information literacy instruction, reference, and library technologies are represented in Figure 4.

In comparison to one another, the three journals bore much similarity in the following ways. Each of the journals carried assessment studies in all topical areas of the current investigation: JAL published 59, portal 23, and CRL 44. The number of information literacy studies ranged from 36 in JAL, to 11 in portal, to 23 in CRL. General satisfaction assessments were minimally covered, with six in JAL, one in portal, and two in CRL. Reference service studies ranged between eight in JAL, six in portal, and eight in CRL. Technology studies saw slightly more coverage than reference service studies, with nine in JAL, five in portal, and 11 in CRL.

Some unique publishing characteristics are noted. JAL covered the largest number of qualifying studies, while also taking the lead in the number of information literacy studies. JAL also published studies from a wider international array of sites, extending to all continents except South America. An illustration of countries represented outside the United States published in JAL appears in Figure 5. The qualifying studies from CRL represented work at sites in the United States, but also in Canada, China, and Norway. The qualifying studies in portal were conducted only in the U.S. and Canada. When viewed in composite, excluding studies from U.S. sites, studies from Australia, Canada, and China were published with greater frequency than the other contributors. (See Figure 6).

Sites where the selected studies took place demonstrate an inclusion of international participation in these primarily American, English-language journals. JAL published the widest variety of international studies, originating in North America, Europe, Africa, Australasia, and Asia. Three information literacy assessments were accepted from Australia, and three reference service assessments came from sites in China. Canada and New Zealand each contributed two information literacy studies, with the remainder of non-U.S. sites each producing one qualifying article in general library satisfaction, information literacy, reference service, and technology (see Figure 5). The majority of studies in JAL naturally originated in the U.S., with four general assessments, 26 on information literacy, five on reference, and six technology studies.

By contrast, CRL and portal published notably fewer studies from sites around the globe. Qualifying studies in CRL came mainly from the United States, with much less representation from Canada (four), China (one), and Norway (one) (see Figure 6). Portal’s publication of studies beyond the U.S. was limited to Canada (two). These numbers are not necessarily a reflection of a publication bias, but rather are the findings based upon the criteria used to select the assessment studies.

5. Discussion

Studies on information literacy dominate the assessment literature in each of the three journals. Mahraj [25] reported parallel findings in her 2006–2011 content analysis of Reference Services Review (RSR) articles. Over the six-year publication spread, 49% of articles covered information literacy topics, 28% reference service topics, and 18% emerging technologies topics. In 2015, Clark [26] continued the retrospective work begun by Mahraj, tracking RSR topical trending between 2012 and 2014. An important difference between the Mahraj and Clark studies and the current research is that methodologically, the former categorized according to broader topical areas, whereas the current study categorized topically but investigated assessment studies exclusively. Relevant findings of Mahraj, Clark, and the current study are compared in

Table 1. Comparison of Findings.

Study & Publication	Information Literacy	Reference	Technology
Mahraj (2006–2011)
RSR	49%	28%	18%
Qualifying articles	119	68	43
Clark (2012–2014)
RSR	47%	12%	30%
Qualifying articles	54	14	35
Current study (2012–2016)
JAL	61%	14%	15%
Qualifying studies	36	8	9
Current study (2012–2016)
portal	48%	25%	27%
Qualifying studies	11	6	5
Current study (2012–2016)
CRL	52%	18%	25%
Qualifying studies	23	8	11

.

Mahraj’s and Clark’s findings of a proportionately greater amount of topical article content about information literacy parallels the findings of the current study and corroborates the findings of Hufford [20], who noted an emphasis upon student learning library assessments in the post-2005 literature. The findings of the current study confirmed this trend. The emphasis upon information literacy and learning outcomes within the literature is easily understood in light of library units’ interest in aligning research agendas to the institutional mission.

The number of reference service assessments in the current study is small. JAL’s six studies, portal’s seven studies, and CRL’s eight studies within this time frame might indicate a change in the role of reference services within academic librarianship. That traditional reference service is in decline is also evidenced in non-research institutions of two- and four-year colleges, as highlighted in a study by Davies and Thiele. Their 2013 study examined article content from Community & Junior College Libraries and College & Undergraduate Libraries published between 2008 and 2010. The report found that among 97 published articles, there was “a dearth of articles published pertaining to reference questions”, and that this phenomenon may have been attributable to a change in “professional focus toward less reference oriented job tasks” [35] (p. 8).

VanScoy and Fontana [36] are of the opinion that the downward trend of published studies in this area may be due to a change in program evaluation needs. As such, new forms of reference, whether embedded within courses, moved to ubiquitous and virtual “spaces”, or tied to information literacy instruction, endure. Reference services, despite competition from popular search engines, continue to meet student needs. The innovations that accompany these new forms must remain relevant, and so are subject to continuous assessment. However, as the findings in the current study indicate, published reference service assessments remain under-represented in top LIS journals [25,37].

The challenges of keeping pace with the latest technological innovations have had a profound effect on libraries. In the present study, only 25 assessments on library technology were identified. The presence of technology studies in portal and CRL, at 27% and 25%, respectively, approach the 30% level discovered by Clark, which may suggest an ongoing increase in the number of technology assessments; however, more data are needed to corroborate this observation. Websites, new and existing software programs, devices, and equipment undergo continuous development, and directly support student and faculty work. The technological future of libraries envisioned by Noh in 2015 is one that follows the progression of web development. She posits that “Library 4.0 must include not only software-based approaches but also technological development such as makerspace, Google Glass, context aware technology, digitization of contents, big data, cloud computing, and augmented reality” [38] (p. 791). While assessment studies of emerging and existing library technologies are not plentiful in the three journals investigated here, they are found within the issues of more specialized library technology publications (e.g., Information Technology and Libraries).

Further Considerations

The current study distinguished 126 criteria-qualifying assessment studies out of 649 articles, drawn from three publications of wide aims and scopes through a labor intensive method. As the authors examined the selected studies, it was observed that, for the most part, findings tended to inform researchers of factual conditions that might eventually lead to improvements. Some studies affirmed that interventions were successful and generalizable; some stated that results were not transferable to other library situations. What was not observed were studies that presented the full assessment research cycle—that is, the implementation of data-driven decisions from initial findings that were then once again assessed. This observation may be the result of a cursory reading of the body of literature examined, but the overview points to a prominence of study findings that potentially lend themselves to implementation and further study.

One of the challenges to the use of empirical evidence in support of decision making is the research-to-practice gap [39]. However, this is being mitigated in publications such as Evidence Based Library and Information Practice (EBLIP), which provides evidence summaries, Health Information and Libraries Journal (HILJ), and journals published by the Emerald Group Publishers (e.g., Performance Measurements and Metrics), which provide structured abstracts, or in the case of HILJ, both a structured abstract and “key messages” for quick perusal. Systematic reviews for the purpose of improving customer service are another evidence resource for manager and practitioner decision making. There is a recently published systematic review for information literacy instruction outcomes [40], but there appeared to be a gap in the literature when the authors searched for current systematic reviews of reference service or holistic library technology assessments. The appendices below identify studies in each of the current study’s assessment study focuses. These may be useful as an index to librarians seeking lists of assessment studies from high-quality publications. The lists may also have the potential to help build the empirical database envisioned by Koufogiannakis [19].

A limitation of the current study is that it examines only three prestigious LIS journals, with fairly wide topical coverage. The limitation to three American LIS journals may have biased the findings of the study and, as a consequence, the findings may be less generalizable. Future research might employ more liberal criteria, such as multi-site studies, or extend data collection to additional journals published outside the United States. Additionally, the study could be expanded to include coverage of other service concerns—for instance, interlibrary loan and document delivery, collection development, access services, technical services, facilities, or non-traditional innovations.

6. Conclusions

Published assessments on general patron satisfaction, information literacy instruction outcomes, reference service, and library technologies at academic institutions were the chief focus areas of this study. The coverage of assessments in these areas by three well-recognized LIS journals points to an ongoing trend toward the continuous improvement goals for satisfied patrons and heightened perceptions of library value. The initial problem for this study was that published assessments of library services in LIS journals concentrated heavily on the coverage of information literacy instruction studies, leaving assessments of other key service areas sparsely covered. The study identified 59 studies in the Journal of Academic Librarianship, 23 studies in portal: Libraries and the Academy, and 44 studies in College & Research Libraries that satisfied the defined criteria. The findings also suggest a growing culture of assessment in many libraries by virtue of the abundance of assessments within the literature.

The review of the literature revealed a second problem: assessment research results are not presented in LIS publications such that they are easily consumable for those who need to make evidence-based decisions. Academic library practitioners seeking to improve services or customer satisfaction measures can turn to an external knowledge base, consisting in part of many library and information science journals. These sources contain empirical studies, which may lead to insights and potential solutions to problems. However, due to research agendas favoring information literacy assessments or studies on other front-line services, innovations in journals with wide scopes are published less frequently. A knowledge base is already large and growing, but it is not as accessible to busy library managers. As a partial solution to this problem, some LIS journals have been reporting evidence summaries that provide structured abstracts and key messages beneath article titles, but this publication practice is not currently as widespread as it might be.

Evidence-based assessments examining general satisfaction and specific front-line services within the three LIS journals investigated in this study demonstrate that librarians within cultures of assessment are actively contributing to the knowledge base. This is seen primarily in the area of information literacy instruction, but secondarily in reference and technology services, as well as general satisfaction studies. These findings represent a snapshot of the depth of data available to decision makers in order to advance their mission of providing valuable library services to all patrons, while also demonstrating value to stakeholders, extending to administrators of the parent institutions.

These findings suggest that editors who publish journals of wider aims and scope might consider the inclusion of a greater number of assessments on services beyond information literacy instruction, and develop innovative models for reporting evidence summaries. To library managers and administrators, the findings suggest that encouragement might be given to staff in all areas, furthering cultures of assessment in a manner that expands research and publication agendas.