| Source: Sarena D. Seifer and Stacy Holmes, Community-Campus Partnerships for
Health,
May 2002 Updated: May 2005 | ||
 | ||
 | Introduction | |
| As with any pedagogy and/or program, a variety of tools and methods can be used to evaluate service-learning. What follows are a discussion of issues to consider, a summary of websites, and a list of publications that provide background, tools, and resources. Remember, new tools and resources are being developed all of the time-use the resources below to get started and check your favorites for updates on a regular basis. | ||
| Issues to Consider | |
| Below are a number of questions that you might consider as you search for the most appropriate tools and resources to evaluate your service-learning program. Clarifying these issues from the beginning of your program will allow you to implement a process that accurately reflects your evaluation objectives and makes the best use of everyone's time and resources. | ||
| Click here to find helpful tools/checklists for planning evaluations on the Evaluation Center's checklist resource page. | ||
| Who is this information for and what do they want to know? | ||
| Are you reporting program evaluation information to a private foundation that provided funds for your service-learning program or are you undertaking a research study with the goal of submitting the findings on student learning to a peer-reviewed journal? Although these two situations might require some similar data, each audience most likely has a different set of standards and needs. For example, a private foundation might be more interested in the number of service hours and the impact on the community, whereas the journal's reviewers might want to know about your research design and significance of change between pre & post tests of student knowledge. The best evaluations fall short when they do not deliver the type of information desired by the intended audience. Implementing a participatory evaluation process can help to alleviate this concern by involving relevant stakeholders early on. Funders, community site staff, faculty, campus administrators, and students can each bring a different set of strengths to the evaluation process and will help to clarify the issues that you should focus on for your evaluation. | ||
| Helpful tools and resources: Michael Quinn Patton's Utilization-Focused Evaluation Checklist Participatory Evaluation: What is it? Why do it? What are the challenges? | ||
| What are you trying to measure? | ||
| Are you interested in student learning objectives or student attitudes towards their surroundings? Do you want to measure the impact of service-learning on community agency partners or the changes within the academic institution? You might have several evaluation goals and each of those goals might have several measurable indicators. For example, an evaluation goal might be to enhance your students' cultural competency. Measurable indicators for this goal might include an increase in knowledge about the demographic composition of the city and state, an increased appreciation for the beliefs and values of other cultures, and an ability to describe strategies for increasing the accessibility of city services to recent immigrants from Mexico who do not speak English. Another measurable indicator may be performance on a reliable and valid tool designed to measure one's cultural competency. | ||
| Clarifying both the goals and the indicators from the beginning of your program will help you determine if you are measuring what you think you are measuring. For example, a change in student attitude does not necessarily indicate increased competency in the course topic. However, attitudinal change might be the primary interest of both you and your service-learning stakeholders (funders, deans, community groups), and as a result the most appropriate focus of your evaluation efforts. | ||
| The following is a list of outcomes that could be impacted by service-learning and therefore evaluated. This list is not comprehensive, but is intended to illustrate a range of outcomes that service-learning has been shown to influence. | ||
| ||
| How will you collect the data? | ||
| As you think about your goals for an evaluation and start to research what other people have done in their service-learning programs, you will undoubtedly note that there are several data collection methods available. It is a good idea to plan this process prior to implementing your service-learning program so that data collection processes can be built into the overall program. For example, information collected on administrative forms filled-out by your students and community partners can become part of the data you use in your evaluation. One useful approach is to connect evaluation to already planned for events and activities - for example, adding questions to an existing course evaluation form or asking community partners to fill out a brief pre- service-learning survey when they attend an orientation. Community site visits can be built into the process as a way to not only keep the program running smoothly but also act as a source of qualitative data collection; for example, through structured interviews or focus groups. The extensive array of evaluation literature and service-learning research studies offers a whole spectrum of methods for collective evaluative information, such as: | ||
| ||
| How will you analyze your data? | ||
| After you have collected your data using one or several of the methods mentioned above, your next step will be to analyze your data. Several issues to consider include: | ||
| ||
| How will you use and disseminate your data? | ||
| Once you have the results in hand, remember to share them with your intended audience and the stakeholders that were instrumental in your evaluation (community partners, students, etc). Also, it is important to let the deans and administrators of the academic institution know the evaluation outcomes. Evaluation findings can be shared in a number of formats, depending on the audience and the goal. For example, you might consider: | ||
| ||
| How will you protect confidentiality and rights of human subjects? | ||
| Generally speaking, routine course and program evaluations do not require institutional review board (IRB) review if they are to be used internally and not for publication. However, we advise checking with your institution's IRB to be sure. Whether IRB review is required or not, it is important to carefully consider how the evaluation will protect the confidentiality and rights of participants. For example, focus groups should begin with a set of "ground rules" that include a commitment by all participants to not divulge any information discussed with others outside the room. Any form for participants to indicate their contact information for receiving the final evaluation report should be separated from the evaluation survey itself. | ||
| Getting Started: General Evaluation Resources | |
| The following list of resources focuses on the basics of evaluation. Most of these websites include links to many other useful websites and resources. Let us know if you have found a useful resource that should be listed! | ||
| American Evaluation Association | ||
| The American Evaluation Association's mission is to improve evaluation practices and methods; increase evaluation use; promote evaluation as a profession and support the contribution of evaluation to the generation of theory and knowledge about effective human action. The Association's website includes a number of useful links, publication lists, and online handbooks. The association also has Topical Interest Groups (TIGs) that include: Assessment in Higher Education; Collaborative, Participatory, Empowerment Evaluation; and International, Cross-Cultural Evaluation. Visit the TIG website to view a list of useful resources and weblinks. | ||
| The Evaluation Center | ||
| The Evaluation Center's mission is to provide national and international leadership for advancing the theory and practice of program, personnel, and student/constituent evaluation, as applied primarily to education and human services. The Center's website includes links to reports, tools, checklists and evaluation research. | ||
| The Resources for Methods in Evaluation and Social Research | ||
| The Resources for Methods in Evaluation and Social Research web
page is a comprehensive site that lists free resources for methods
in evaluation and social research. This "how-to" resource focuses
on particular methods of evaluation including: surveys, focus
groups, sampling, interviews, and other methods. Extensive lists of
links to other useful resources. This site is compiled by Ya-Lin
Liu and Gene Shackman. Abstract source: WWW Virtual Library Evaluation. | ||
| The Online Evaluation Resource Library | ||
| The Online Evaluation Resource Library (OERL) was developed for professionals seeking to design, conduct, document, or review project evaluations. OERL is funded by the National Science Foundation (NSF) and includes tools and sample reports covering such topics as curriculum development, student attitudinal assessment, access for under-represented populations, and faculty development. | ||
| Practical Assessment, Research and Evaluation | ||
| Practical Assessment, Research and Evaluation (PARE) is an on-line journal supported, in part, by the Department of Measurement, Statistics, and Evaluation at the University of Maryland, College Park. Its purpose is to provide education professionals access to refereed articles that can have a positive impact on assessment, research, evaluation, and teaching practice, especially at the local education agency (LEA) level. | ||
| The WK Kellogg Foundation | ||
| The WK Kellogg Foundation (WKKF) website offers the WKKF
Evaluation Handbook and a variety of other useful publications,
including: Methods of Assessing the Quality of Public Service and Outreach in Institutions of Higher Education How Service Works - A Kellogg evaluation of higher education service programs | ||
| Getting Started: Evaluation Resources Specific to Service-Learning | |
| The evaluation resources listed here are specific to service-learning programs and/or service-oriented programs that can act as models. Most of these websites include links to many other useful websites and resources. Let us know if you have found a useful resource that should be listed! | ||
| The Community-Campus Partnerships for Health | ||
| The Community-Campus Partnerships for Health (CCPH) website
includes tools to assess service-learning and students, faculty,
community partners and institutions, many of them excerpted from
the publication Shinnamon A, Gelmon S and Holland B. Methods and
Strategies for Assessing Service-Learning in the Health
Professions. San Francisco, CA: Community-Campus Partnerships for
Health, 1999
(click here for ordering information) Methods and strategies for student assessment Methods and strategies for community partner assessment Methods and strategies for faculty assessment and reflection Methods and strategies for institutional assessment Self-assessment tool for service-learning sustainability | ||
| The National Society for Experiential Education | ||
| The National Society for Experiential Education website includes a list of publications and resources, such as information on the Program Evaluation Handbook by Robert Serow (written specifically to introduce experiential educators to the basic elements of program evaluation) and the volumes of the Combining Service and Learning: A Resource Book for Community and Public Service series which includes a section on evaluation. | ||
| The Shumer Self-Assessment for Service-Learning (PDF, 122K) | ||
| The Shumer Self-Assessment for Service-Learning (SSASL) is designed as a self-reflective system for professionals in the service-learning and experiential learning fields. Although most applicable to K-12 service-learning programs, it offers a series of instruments and analysis worksheets arranged to help individuals evaluate and improve their current service-learning programs. (Copyright December 2000 by the Center for Experiential and Service-Learning, Department of Work, Community, and Family Education, College of Education and Human Development, University of Minnesota, St. Paul, Minnesota) | ||
| University of Southern California's Joint Educational Project (JEP) | ||
| Homepage includes a discussion of evaluating service-learning programs, including the evaluation of the student as engaged citizen, student learning, community sites, and the individual course/program. | ||
| The Compendium of Assessment and Research Tools | ||
| The Compendium of Assessment and Research Tools (CART) is a database that provides information on instruments that measure attributes associated with youth development programs. CART includes descriptions of research instruments, tools, rubrics, and guides and is intended to assist those who have an interest in studying the effectiveness of service-learning, safe and drug-free schools and communities, and other school-based youth development activities. | ||
| Campus Compact | ||
| Campus Compact has a variety of resources, including publications that cover the evaluation of service-learning and other campus-based programs. Examples of publications that are listed on their website include: | ||
| Assessing Service-Learning and Civic Engagement: Principles and Techniques. A comprehensive guide for planning and implementing assessments of service-learning and civic engagement programs. Provides a clear assessment model, specific examples of measurable indicators, and many different types of qualitative and quantitative instruments for assessing program impact on faculty, students, communities, and institutions. Revised, third edition. Sherril B. Gelmon, Barbara A. Holland, Amy Driscoll, Amy Spring, Seanna Kerrigan. 2001 | ||
| Evaluation and Assessment in Service-Learning. A resource for service-learning practitioners who want to improve their programs and to assess the short- and long-term impacts of their programs. Includes student pre- and post-tests, surveys, and resources. 1997. | ||
| Integrating Service with Academic Study Assessing Internal and External Outcomes of Service-Learning Collaborations. Provides critical insights from community college and university faculty for enhancing and improving service-learning programs. From institutional self-assessments to assessing the community impact, this sourcebook provides the tools necessary to improve service-learning programs. 1998. | ||
| RAND | ||
| RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. The RAND website includes several evaluation reports that explain the methods and tools used in their research, such as their 1999 report on the Learn and Serve community service project. The report can be downloaded via the RAND website: | ||
| Combining Service and Learning in Higher Education: Evaluation of the Learn and Serve America, Higher Education Program. Maryann Jacobi Gray, Elizabeth Heneghan Ondaatje, Ronald D. Fricker, Sandy A. Geschwind, Charles A. Goldman, Tessa Kaganoff, Abby Eisenshtat Robyn, Lori J. Vogelgesang, Stephen P. Klein, Nancy Campbell, Kathy Rosenblatt. (1999). This report presents results of a three-year evaluation of Learn and Serve America, Higher Education a program sponsored by the Corporation for National and Community Service that aims to increase involvement in community service by higher education institutions and students. | ||
| The Evaluation Handbook: Practical Tools for
Evaluating Service Learning Programs, 1997 (Service Learning 2000 Division of Youth Service California) | ||
| Down-to-earth and realistic, The Evaluation Handbook
demystifies every step of the evaluation process. Written for
anyone who contributes to the evaluation of a service learning
program-teachers, coordinators, administrators, parents,
community-based staff-the Handbook offers a wealth of field-tested
examples, tools and strategies.
($25) (Abstract source: Service-Learning 2000 website) | ||
| The Corporation for National and Community Service | ||
| The Corporation for National and Community Service has several evaluation-related resources that can be found through their main website. Several of the technical assistance and training manuals for service programs such as AmeriCorps and Senior Corps have "how-to" chapters on evaluating components of the service programs. Check-out the example listed here and search the main website for more information. http://nationalserviceresources.org/files/legacy/filemanager/download/571/TB.15.pdf | ||
| Project STAR | ||
| Project STAR, a Corporation for National and Community Service sponsored technical assistance provider in evaluation, can assist National Service programs with evaluation endeavors. The Project STAR webpage links to information specific for Learn and Serve Higher Education programs, Learn and Serve K - 12 and Community Based programs, and Community-Higher Education-School Partnership projects on how to develop and conduct a program evaluation. | ||
| The Effective Practices Information Center (EpiCenter) | ||
| The Effective Practices Information Center, or EpiCenter, is the online source of effective program practices sponsored by the Corporation for National and Community Service. The mission of EpiCenter is to support the development of sustainable programs by sharing what works in community service and volunteering. You can search their online database of effective practices by keywords, including topics such as assessment and evaluation. | ||
| The Community-Campus Partnership for Health (CCPH) | ||
| The Community-Campus Partnership for Health (CCPH) website includes a service-learning bibliography with a section on Evaluation Methods. | ||
| Additional Links on Assessment and Evaluation | ||
| The NSLC links section includes additional links on Assessment and Evaluation (some overlap with this Fact Sheet). | ||
| The Evidence Base for Service-Learning in Higher Education | ||
| For information on the evidence base for service-learning and published research, visit the NSLC Fact Sheet entitled "The Evidence Base for Service-Learning in Higher Education." Information provided in that fact sheet is relevant to service-learning evaluation as well. | ||
