当前位置:首页 >> IT/计算机 >>

From input and output measures to quality and outcome


From input and output measures to quality and outcome measures, or, from the user in the life of the library to the library in the life of the user
By Martha Kyrillidou1

(this is the pre-print version; the final print version appeared in the Journal of Academic Librarianship, volume 28, number 1, pages 42-46)

Abstract: This paper discusses library models and user roles in relation to the transformative effects of technology. It explores how these models are changing our conceptualization of library assessment models from a linear to a more cyclical and dynamic perspective shaping relations between input, output, quality and outcome indicators. "Nothing is permanent but change" - Heraclitus Do libraries matter in the life of the everyday men and women, in the lives of the readers who explore new ideas and thoughts, in the lives of the scholars who are engaged in the systematic investigation of furthering knowledge? These may be rhetorical questions; yet, their answer may be evading the reality of higher education institutions where the value of the library is sometimes placed at a peril in an era of competing priorities, limited resources, and exponential growth in the areas of technology, information content, and communication capabilities. In a recent invited address at the Library Research Seminar II, Yvonna S. Lincoln captured the audience's imagination by providing insights into the perception of library services and users from qualitative research. In her remarks "library service is not performed … but enacted as an elaborate cultural ritual, the texture and fabric of which is changing in front of us ..Embodying multiple overlays of meaning … many of which might have to be discovered - or even created."2 The need to create the new library, to reinvent or reconceptualize, is real and many are engaged in trying to do just that.3 Possible models to emulate are available by looking either backward, learning from the historical perspective, or looking forward, learning from newly gained insights into the operation of future libraries. Libraries have adopted different pragmatic and symbolic historical roles throughout the years. Krummel describes the historical perspective of libraries in terms of the seven ages of librarianship. In his seven ages of librarianship Krummel used as a criterion for determining the different ages what he calls "the objectives" of the libraries, which are characterized, as quotidian (around 3000 BC), intellectual (around 300 BC), religious (around 500 AD), virtuous (around 1350 AD), scientific (around 1600 AD), utilitarian (around 1840 AD), and pragmatic (around 1910 and beyond). Each one of these adjectives describes the different messages libraries sustained throughout the ages. 4 Peter Brophy also tries to describe possible future models for libraries in the 21st century. He

1 of 8

sees a number of forces shaping this future ranging from the information plethora, the unstable information objects, the lack of quality assurance of the information content, the heterogeneity of technical service standards, the development of metadata in a haphazard and semantically confused way, users thinking not needing intermediaries and client access being pervasive and universal. He sees five models for the future, in which library serves as: a physical presence, as a memory institution, a learning center, a community resource, and invisible intermediary.5 Where is the momentum for examining new and old models coming from? To some extent from the transition from a model where people where coming to the library to get their information to a model where the information and the library is actively engaged in taking the resources to the user, a movement towards disintermediation facilitated by technological developments. This movement has been so successful that it recently made the pages of the Chronicle of Higher Education as a concern "about the future of what has traditionally been the social and intellectual heart of campus, as well as about whether students are learning differently now;" some though are less alarmed and more encouraged that more and more materials are online resulting in less and less use of the brick-and-mortar library accepting the fact that libraries will become more virtual without seeing it as a big loss.6 The library of the future is less of a place and more of a portal that provides high quality content on the Web - a scholar's portal for research libraries.7 Access to digital materials though involve challenges of "scale, funding, law, and access"8 as well as issues of cultural change and transformative assessment.9 The driving force behind disintermediation is in part the need to make information dissemination more effective and efficient and technology has facilitated this to a large extent. Historical strategies that libraries have adopted to achieve this goal include the establishment of open stacks, user initiated ILL/DD, document delivery to the faculty and student offices, full-text delivery at the desktop, virtual reference services. The dislocation of the library from the role of "central control" where information is acquired, organized, and disseminated to a role of "facilitated access" without necessarily fully controlling acquisition, organization and dissemination of the information content has already happened through outsourcing, licensing, alternative publication channels, and increased user control (see Figure 1). In the 'good old days' it was generally acceptable to justify the existence of the library in terms of the extent of resources available emulating the Alexandrian ideal. As early as the beginning of 19th century though Balbi argued "1,000 volumes of a certain kind in a certain place might be of more value than 20,000 or 30,000 of another kind or in another place."10 This ultimate goal of bringing together a perfectly customized collection of books for the purposes of fulfilling users' needs was driving collection size constantly upwards in the 20th century. It is really libraries' commitment to quality that was driving quantity; librarians are concerned to throw away information content and want to safeguard the container, traditionally known as the book. Librarians are concerned that they may be throwing away the missing piece in a puzzle which as far as we know may never be set up. The reinvention of the Library of Alexandria in its digital manifestation is still inspiring to digital library developers who see technology facilitating the goal of universal access to our cultural heritage. 11 The value of a library's collection as expressed in terms of magnitude of resources was not into question though till recently. Technological innovations are making possible the decoupling of information content from the container. The container, especially computing storage and processing power, are depreciating fast while we are facing the challenge of understanding and

2 of 8

demonstrating the conditions under which information content appreciates. Thus, libraries are being called to prove themselves in terms of the value provided to the users in ways that can be described and potentially measured. Libraries are social institutions being part of the social capital available to a community. As such their value needs to be articulated in relation to the value they provide to the user, for the user and by the user. Another reason why it is increasingly inadequate to describe libraries only in terms of the extensiveness of resources is that control of such resources is increasingly residing outside library channels. Assessment efforts for developing measures for the networked environment12 and digital libraries need to take into consideration the collaborative environment that needs to be in place among authors and creators of information content, distributors and publishers, as well as libraries.

MEASUREMENT MODELS
Libraries have been recognized for succeeding in measuring themselves in terms of input, and more recently in terms of output measures. Inputs are the resources available to the system, ranging from financial, staffing and material resources in analog or digital forms; outputs are the activities the system exports ranging from transactions, to hours the premises are available, to the availability, use and usability of the material resources to name a few. Library input measures and statistics for research libraries 13 go back to 1908 and output measures have been collected for the most part of the 1990s. However, for the most part of the 20th century libraries did not engage systematically in measuring quality or outcomes from a user perspective. Although there have been efforts to measure the effects of the use of resources, these efforts often were done in the context of local considerations as large scale cooperative and coordinated assessment activities were more difficult to develop in the absence of recent technical and communications capabilities. Research libraries only recently have been able to move from inputs and outputs into some systematic evaluation of quality of services across institutions at a macro level.14 A year ago during the ARL Symposium on "Measuring Library Service Quality" the latest thinking in relation to measuring library service quality met in one room resulting in a rich exchange of ideas about the theoretical implication and practical consequences of measuring library service quality.15 As librarians just barely manage to come to some understanding of quality issues, accreditation agencies are knocking at the door of the various institutions with questions regarding outcomes. So what is the relation between inputs, outputs, quality and outcomes? Unfortunately, we still know very little about the relation of these constructs. We were not focused on measuring outputs till about a decade ago, and only in the last few years we have engaged as a community the measurement of quality from the user's perspective. The best we can currently do is examine local efforts regarding the measurement of outcomes in libraries and develop the groundwork that will give birth to renewed understanding in this area over the next few years. There has been extensive work in higher education in relation to outcomes assessment for a couple of decades now; more recently some major empirical studies gained prominence, yet their long-term impact is yet to be realized.16 This paper presents three different visual models for thinking in terms of inputs, outputs, quality and outcomes from a library and a user perspective. Often there is an implicit simplistic assumption that inputs have a direct relation to outputs, which in turn relates to quality and to outcomes (Figure 2). Unfortunately, there is little

3 of 8

empirical evidence that would support these assumptions. We know for a fact that input measures such as volumes held, serial subscriptions, expenditures, and staffing do not relate to output measures as currently measured in terms of reference transactions, circulation, interlibrary loan, bibliographic instruction. One of the reasons for this lack of an observed relation in the current ARL Statistics, for example, is the multiplicity of special local arrangements, restrictions, opportunities and circumstances that inflate the measurement error into proportions that make output metrics across institutions meaningless but for some gross generalization that 'libraries circulate books as can be observed from the fact that they can all report circulation statistics.' However, there is not complete absence of a relation, as we know that increased availability of resources affects their use from the observed relation that exists between the size of the collection and circulation. Furthermore, we know that the current empirical evident does not show a strong relation between quality constructs as measured by users perceptions and expectations in relation to input or output measures, i.e. perceived quality as judged by the user does not relate to the extensiveness of resources or activities in a library. The relation between these constructs is dubious partly because library procedures and processes have been institution centric; library procedures and policies are in place to safeguard the effective operation of the library primarily and secondarily to facilitate the effective use of resources by users. The relation depicted in Figure 2 is also dubious because it tries to simplify and depict in a linear way a progression of constructs that may be related in a more complex fashion more like the diagram depicted in Figure 3. Input measures such as books are clearly the output of users who as scholars and authors have created these resources which result into increased income, reputation or recognition enhancing personal goals and outcomes. Another example for understanding the multidimensional nature of user behavior is the typical graduate student. A person who is struggling to complete his/her dissertation and uses information resources in a multiplicity of ways, from directly borrowing library resources, to talking to faculty members who have used the library themselves, to borrowing a book from his/her advisor, to finding another interesting resource while visiting a fellow-students' home, to communicating via email with other experts who refer him/her to additional resources and citations, to browsing the internet and locating chats and discussions in his/her specific field of interest, to sending reference inquiries to ask-a-service operations, to talking with a fellow student whose spouse works in the library and discusses with him/her the articles he/she was able to locate in the library, or with the help of a librarian, to sharing a draft of his paper with colleagues, etc. Every one of these transactions has multiple dimensions of inputs, output, and quality and outcome elements from multiple interactive and reflective perspectives. This inextricable and complex web of relations provides a richer context for the usefulness of inputs, outputs, quality issues, and the impact libraries are making. The usual linear model (Figure 2) depicting a linear, sequential, rational, and controlled set of relations is indeed limiting our understanding of a model (Figure 3) that is more cyclical, haphazard and uncontrolled. This becomes even more complex when one tries to introduce the notion of motion (Figure 4) depicting a more dynamic and flexible model, moving users and information resources into a spiral swirl up and down into the depths of knowledge, exploration and experience.

DEFINING OUTCOMES
In addition to being able to work with more dynamic models in describing libraries and users,

4 of 8

we lack an adequate shared understanding of how outcomes are defined. The Institute of Museum and Library Services defines outcomes as "benefits or changes for individuals or populations during or after participating in program activities, including new knowledge, increased skills, changed attitudes or values, modified behavior, improved condition, or altered status (e.g., number of children who learned a finger play during story time, number of parents who indicated that they gained new knowledge or skills as a result of parent education classes, number of students whose grades improved after homework clinics, number of children who maintained reading skills over the summer as a result of the summer reading program, number of people who report being better able to access and use networked information after attending information literacy classes). While outcome measurement may at first seem very different from the traditional program or service model, in fact it incorporates all of the elements of traditional library measurement (inputs, activities, outputs) while adding only the element of outcomes."17 The National Postsecondary Education Cooperative (NPEC) is also sponsoring work in the area of defining student outcomes: "The Student Outcomes-Common Language project will reconcile the meanings and relationships of multiple terms used to describe student outcomes (e.g., performance, competency, student learning, skills, abilities, institutional effectiveness, assessment, quality assurance, and indicators), and develop definitions and concepts that support more effective communication and use of information about student outcomes."18 Within the ARL New Measures agenda there are at least four implied definitions of outcomes in terms of how various working groups and projects are approaching the issue: (a) learning outcomes, (b) research outcomes, (c) institutional outcomes, and (d) personal control or electronic service quality issues. Learning outcomes is a larger area of investigation where educational research and assessment faculty have been engaged in trying to identify how students can achieve their learning objectives more readily. Within libraries one way for approaching learning outcomes is through the ACRL Information Literacy Standards Effort where a formal instructional program is advocated as a way of reaching information competency levels. The ARL Working Group on Learning Outcomes is taking a broader view building upon work proposed in Ken Smith's white paper19 and currently following the NPEC work mentioned earlier, surveying accreditation agencies and identifying institutions with exemplary outcomes assessment programs, exploring what other academic and learning assessment professional associations are doing, and investigating national surveys currently being conducted on campuses. Another area of investigation within the ARL New Measures Initiative focuses on research outcomes. Carolynne Presser developed a preliminary sketch20 and further investigation is currently underway by Doug Jones, ARL Visiting Program Officer. Also related is Brinley Franklin's investigation of the relation between the extent to which academic libraries support sponsored research reporting data from studies conducted between 1982 and 2001 at 153 libraries incorporating responses from approximately 150,000 academic library users. His findings indicate a "high correlation between total research and development funding at an educational institution and total library expenditures at research universities. Little or no relationship was determined, however, between total library expenditures and library expenditures in support of sponsored research as a percentage of total library expenditures."21 Both learning and research outcomes may be viewed from a larger perspective of institutional effectiveness. Institutional outcomes are also briefly discussed as part of the ARL E-Metrics Phase II report.22

5 of 8

Researchers and library users want to have more personal control over the resources they access. Yet personal control can be both a barrier to access as well as an enabler. Research trying to identify the personal characteristics that help people readily adopt technology is one way to judge readiness,23 but it will not safeguard us in the short term from failing to locate information that may not be available electronically. The relation between the quality of electronic library services and outcomes is yet another challenging issue where more research is needed.24 Many digital library projects are engaging in some form of evaluation or another, but there is also some acknowledgment that evaluation is not yet seen as an integral part of the development cycle built tightly into the ongoing operation of a digital library and linked to the transformation of the cultural context. Developers perceive evaluation as something to be done after development is complete, and evaluators often perceive development from the perspective of a finite assessment cycle. Yet evaluation is an integral part of the critical thinking processes we engage in, helps adjust our course of action, and it is taking place all the time consciously, or unconsciously - it is our responsibility to be ready to learn and interpret the developing context all the time.

Footnotes
1 Senior Program Officer for Statistics and Measurement, Association of Research Libraries, 21 Dupont Circle, Washington DC 20036 <martha@arl.org>. 2 Yvonna S. Lincoln, "Insights into Library Services and Users from Qualitative Research" Invited Address for the Library Research Seminar II, University of Maryland-College Park, Maryland, November 2-3, 2001. 3 William Y. Arms, Digital Libraries (Cambridge, Massachusetts: MIT Press, 2000). 4 D. W. Krummel, Fiat lux, fiat latebra : a celebration of historical library functions (Champaign, IL: Publications Office, Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign, c1999). 5 Peter Brophy, The Library in the 21st Century : New Services for the Information Age (London: Library Association Publishing, 2000). 6 Scott Carlson, "The Deserted Library: as students work online, reading rooms empty out leading some campuses to add starbucks" The Chronicle of Higher Education. 7 Jerry D. Campbell, "The Case of Creating a Scholar's Portal on the Web: a White Paper" ARL: a Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC <http://www.arl.org/newsltr/211/portal.html> (accessed November 20, 2001). 8 Brewster Kahle, Rick Prelinger, and Mary E. Jackson, "Public Assecc to Digital Material" D-Lib (October 2001) <http://www.dlib.org/dlib/october01/kahle/10kahle.html> (accessed November 20, 2001). 9 Thomas A. Angelo, "Doing Assessment as if Learning Matters Most" AAHE Bulletin (May 1999) <http://www.aahe.org/bulletin/angelomay99.htm> (accessed November 20, 2001).

6 of 8

10 Adriano Balbi, A Statistical Essay on the Libraries of Vienna and the World (Jefferson, North Carolina: McFarland, 1986), ix. 11 Brewster Kahle, Rick Prelinger, and Mary E. Jackson, "Public Assecc to Digital Material" D-Lib (October 2001) <http://www.dlib.org/dlib/october01/kahle/10kahle.html> (accessed November 20, 2001). 12 Wonsik "Jeff" Shim, Charles R. McClure, Bruce T. Fraser, John Carol Bertot, Arif Dagli, and Emily H. Leahy, Measures and Statistics for Research Library Networked Services: Procedures and Issues: ARL E-Metrics Phase II Report (Washington, DC: Association of Research Libraries, 2001) <http://www.arl.org/stats/newmeas/emetrics/index.html> (accessed November 20, 2001). 13 Martha Kyrillidou, "Research Library Trends: ARL Statistics" Journal of Academic Librarianship (November 2000): 427-436. <http://www.arl.org/stats/arlstat/jal99.html> (preprint version accessed November 20, 2001). 14 Thompson, B., Cook, C., and Thompson, R.L. (in press). "Reliability and structure of LibQUAL+TM scores." Portal: Libraries and the Academy. [out January, 2002]; Cook, C., Heath, F. and Thompson, B. (in press). "Score norms for improving library service quality: A LibQUAL+TM study." Portal: Libraries and the Academy. [out January, 2002]; Heath, F., Cook, C., Kyrillidou, M., & Thompson, B. (in press). "ARL Index and other validity correlates of LibQUAL+TM scores." Portal: Libraries and the Academy. [out January, 2002]. <http://www.libqual.org/> (accessed November 20, 2001). 15 Martha Kyrillidou and Fred Heath (eds) "Measuring Service Quality" Library Trends (Spring 2001). 16 George D. Kuh, "Assessing what really matters to student learning: Inside the National Survey of Student Engagement" Change 33 (3) 10-17, 66. <http://www.indiana.edu/~nsse/> (accessed November 20, 2001); Higher Education Research Institute at UCLA has been engaged into systematic research in this area for over three decades now <http://www.gseis.ucla.edu/heri/heri.html> (accessed November 20, 2001). 17 Institute of Museum and Library Services, Perspectives on Outcome Based Evaluation for Libraries and Museums, p.20 <http://www.imls.gov/pubs/pdf/pubobe.pdf> (accessed November 20, 2001). 18 NPEC Project - Student Outcomes - Common Language <http://nces.ed.gov/npec/sos_common.html> (accessed November 20, 2001). 19 Ken Smith, "New Roles and Responsibilities for the University Library: Advancing Student Learning through Outcomes Assessment" <http://www.arl.org/stats/newmeas/outcomes/HEOSmith.html> (accessed November 20, 2001). 20 Carolynne Presser "Libraries Impact on Research: a Preliminary Sketch" <http://www.arl.org/stats/program/presser.pdf> (accessed November 20, 2001). 21 Brinley Franklin, "Academic Research Library Support of Sponsored Research in the United

7 of 8

States" Proceedings of the 4th Northumbria Internation Conference on Performance Measurement in Libraries and Information Services (Washington, DC: Association of Research Libraries, forthcoming). 22 Wonsik "Jeff" Shim, Charles R. McClure, Bruce T. Fraser, John Carol Bertot, Arif Dagli, and Emily H. Leahy, Measures and Statistics for Research Library Networked Services: Procedures and Issues: ARL E-Metrics Phase II Report (Washington, DC: Association of Research Libraries, 2001): 15-26. <http://www.arl.org/stats/newmeas/emetrics/index.html> (accessed November 20, 2001). 23 A. Parasuraman, "Technology Readiness Index (TRI): a Multiple-Item Scale to Measure Readiness to Embrace New Technologies" Journal of Service Research (May 2000): 307-320. 24 Valarie A. Zeithaml, A. Parasuraman, and Arvind Malhotra "A Conceptual Framework for Understanding e-Service Quality: Implications for Future Research and Managerial Practice" Working Paper, Report No. 00-115 (Cambridge, MA: Marketing Science Institute, 2000). Also, Texas A&M and ARL have recently been awarded a grant for modifying LibQUAL+ for the digital library environment <http://www.arl.org/libqual/geninfo/nsdlpr.html> (accessed November 20, 2001).

ARL Statistics and Measurement Program Home | ARL Home

Maintained by ARL Web Administrator ? Association of Research Libraries, Washington, DC Last updated: Thursday 8 August, 2002.

8 of 8


相关文章:
更多相关标签: