06/01/2013

Learning Outcomes Assessment Step-By-Step: The Story Behind NCDA’s New Monograph

By Shawn P. Conlon

Did you ever wonder how an NCDA monograph makes it to the bookshelves? Here’s your chance to get a glimpse behind the scenes. Experience the energy and passion that spurred the development of one of NCDA’s newest resources.

In this article, Shawn Conlon interviews the monograph authors, Julia Panke Makela and Gail S. Rooney, to learn what difference learning outcomes assessment can make for career development professionals. Panke Makela and Rooney see career development professionals as educators who must assess client learning. They take the field beyond traditional attendance data and client assessments. Their approach identifies change in clients and effectiveness of programs.

Shawn: What stimulated you to think about assessing learning outcomes, and what are the benefits of assessing learning in practice?

Julia & Gail: We found our inspiration for focusing on learning outcomes assessment independently, over a decade ago. Gail was working as Vice President for Student Services at Briar Cliff University in Sioux City, Iowa; Julia was working nearly 1,200 miles away, as the Coordinator of Career Services for the School of Management at George Mason University in Fairfax, Virginia. At that time, academic programs on higher education campuses across the country were experiencing great pressure from accreditation boards and other external bodies to provide evidence of student learning. Student affairs professionals were not under the same intense pressure of our academic counterparts, yet we both found ourselves in environments where administration and faculty encouraged our contributions to outcomes assessment. As we took steps in this direction, campus leaders actively recognized the unique and valuable contributions that we, as student affairs professionals, were making to campus-wide assessment efforts.

Shortly after we began our own explorations into outcomes assessment, the National Association of Student Personnel Administrators (NASPA) and the American College Personnel Association (ACPA) released their groundbreaking learning outcomes and assessment statements for student affairs with the publication of Learning Reconsidered in 2004,¹ that further fueled our enthusiasm. In late 2004, our paths crossed at the University of Illinois where Gail was serving as the Director of The Career Center and Julia was engaged in consulting roles while pursuing her Ph.D.

As we further embraced learning outcomes assessment, we found ourselves intrigued by new questions for our practice. The old questions that had previously driven our data collection efforts in career services included: How many events did we run this year? How many clients participated? How satisfied were they? How many clients found jobs or went on to further education after graduation? While the answers to these questions remain important for informing decisions about our practice, we now started asking: What difference do career services make in clients’ lives? How have clients changed as a result of their interactions with career services? What do they know? What can they do or demonstrate? How do they feel?

The focus of our assessment shifted from counting program components to closely reflecting upon the client’s experience – understanding how clients change during their interactions with career services. Providing evidence that tells the story of transformation that career services can make in client’s lives helps us determine how career services may be improved, and allows us to advocate for the unique ways that we contribute to the educational institutions we serve. As you can see, there is plenty to stimulate our thinking and keep us engaged for many years to come!

Shawn: Tell us about the eight-step ALOI Cycle presented in the monograph.

Julia & Gail: Many career professionals find the prospect of taking on learning outcomes assessment daunting. We often feel that there is not the time or resources to engage in assessment. Or, it may require assessment skills that are beyond our current personal expertise. These challenges can be diminished by taking a step-by-step approach that integrates assessment into daily practice. The Assessment of Learning Outcomes for Interventions (ALOI, pronounced “alloy”) Cycle is an eight-step process for planning and conducting learning outcomes assessments that are tailored to specific client groups and career interventions. The eight steps include:

1) defining context

2) brainstorming outcomes

3) writing outcomes statements

4) connecting theories and professional standards

5) prioritizing learning outcomes

6) evaluating learning outcomes

7) reflecting on results and process

8) using learning outcomes assessments

The monograph provides worksheets with several examples to guide readers through each step. The most effective way to gain comfort and confidence with learning outcomes assessment is to actively apply it to a familiar career intervention, and to do so with the help of colleagues and peers. To begin, we recommend gathering a team of career professionals or other stakeholders with an interest in the effectiveness of a career intervention to join you in the assessment process.

Shawn: What advice do you have for practitioners making their first attempt at assessment?

Julia & Gail: The first key to successful learning outcomes assessment is to get started. In times when you feel overwhelmed, remember: Get started. Start small. Build up, brick by brick. Think of assessing learning in terms of building a house. Focus on laying one brick at a time, no matter how small, and building up a strong foundation — a rich body of evidence.

Begin with small projects associated with career interventions that you believe to be successful. Test your assumptions, and see what you can discover. Explore how a small learning outcomes assessment project can tell the story of how career services makes a difference in clients’ lives, and look for ways to build upon and enhance your story. Small, positive experiences with learning outcomes assessment can teach useful skills, build confidence and capabilities, and motivate future learning outcomes assessment efforts.

Shawn: Can you describe the monograph in a "bumper sticker" phrase or two?

Julia & Gail: Discover how your career programs and services make a difference in clients’ lives. This monograph teaches a practical approach to learning outcomes assessment that helps you tell the story of your career programs and services, celebrate your successes, and continuously improve your practice.

For more information on this monograph, visit the NCDA Career Resource Store or contact Julia Panke Makela at julia.makela@gmail.com

Reference

Keeling, R. P. (Ed.). (2004). Learning reconsidered: A campus-wide focus on the student experience. Washington, DC: National Association of Student Personnel Administrators.

Julia Panke Makela, Ph.D., NCC,is a Research Specialist in the College of Education at the University of Illinois at Urbana-Champaign. She has worked in career services in higher education as a counselor, consultant, evaluator, and researcher for over 12 years. She currently serves NCDA as a member of the Ethics Committee and the Publications Development Committee.

Gail S. Rooney, Ph.D.,is the Director of The Career Center at the University of Illinois at Urbana-Champaign. She has worked in higher education for over 30 years as an administrator, counselor, and faculty member. Assessment has been central to her work in a variety of institutions from small colleges to community colleges and large universities.

Shawn Conlon, Ed.S., NCC, is the Head of the Personal & Professional Development Branch for the United States Marine Corps. He can be reached at shawn.conlon@gmail.com.