Evaluating Adult Education for Organizational Learning
From Adult Education
Created by Billy Skinner, Micheal Neal, & Jennifer Fimbel Fall 2007
Contents |
Introduction
After presenting a program no matter whom the audience may be, as the program coordinator, you are going to evaluate your program to see if there are things you can change. These changes may be as small as adding something minor or taking something out to make your program run smoother. No two individuals will evaluate a program in the same manner. Not only are you going to evaluate the program, but in some way as the program coordinator you will also evaluate co-workers or others that assisted in making the program reality. The evaluation of these individuals is one form of evaluation. In order to have a successful program and to get along with everyone that has input into the program it is vital to have open communication with all team members. Here are some things that may be helpful to facilitate effective communication skills and promote teamwork for the success of the program.
Program Evaluation Defined
According to Carter ( 2007), program evaluation is carefully collecting information about a program or some aspect of a program in order to make necessary decisions about the program. Cafferella (2001) defines evaluation as "a process used to determine whether the design and delivery of a program were effective and whether the proposed outcomes were met" (p. 225). Program evaluation can include any or a variety of at least 35 different types of evaluation, such as needs assessments, accreditation, costs/benefits analysis, effectiveness, efficiency, formative, summative, goal based, processes, outcomes, ect. The type of evaluation you understand to improve your programs depends on what you want to learn about the program.
Types of Program Evaluation:
1. Goal-Based Evaluations - are evaluating the extend to which programs are meeting predetermined goals or objectives.
2. Process-Based Evaluations - are geared to fully understand how a program works, how does it produce that results that it does.
3. Outcomes-Based Evaluation- used primarily for non-profit organizations and its usually asked for by funders. It asks if the organization is really doing the right program activities to bring about the outcomes you believe to be needed by the client.
Ways to collect information:
- Surveys - Interviews - Document Review - Observation - Focus Groups - Case Studies
When Should Evaluation be Used?
Evaluations are excellent tools for the adult educator. We can use the collected data to measure change, staff performance, self performance, absorption of knowledge, adoption of skills, budget of programs, outcomes of programs, and the satisfaction of our audience. We can than incorporate our data and use it as a reporting source for stakeholders, future program plans, the design and methodology for future and existing programs (classes, courses, etc.), and determine what behaviors changed as a result of the program and/or teaching methods.
So when should we plan for evaluation and what does and does not get evaluated? Evaluation should take place from early on in the planning stages of a possible program right on through to the end and beyond.
In the early stages of program development we need to assess the need for a particular program. Our evaluation may be as simple as conducting surveys of individuals who are potential participants, to collecting data gathered from other sources. We must then evaluate the program possibilities as to the budget of the program, stakeholder interest, mission of your work place, and plausibility of organizing the program. Plausibility should include program costs, participant costs, timing (time of year, day of the week, evening versus daytime, etc.). Once the people, place and stuff is put into place, planning your marketing strategy will be crucial to the success of obtaining the target audience. Evaluating your marketing strategies before and after the program can change your perspective on where your marketing budget is best spent for future programs of the same type.
Evaluating your program during the program itself may not be a formal evaluation, but it is important in the overall scheme of success for your participants. An example that happened to me follows.
We had spent nearly a year planning the Small Farm Expo. I attended just to observe as we would be hosting/organizing it the following year. One of the topics featured was raising goats. The speaker contracted with was a successful dairy goat operator. He never showed up. There we were with nearly 75 people in the room waiting for a lecture on raising goats. Now I don't have a whole lot of experience with dairy goats, however I can wing it for meat goats as the similarity to raising commercial lambs is very close. So I EVALUATED the knowledge in the audience, asked some simple questions like, "Why are you here?", "Who would be interested in raising goats for meat and who would like to raise dairy goats?" Thank goodness the majority wanted to raise meat goats and for those who didn't I pointed them to the dairy goat display area where there would be people who could assist them in their venture. My ultimate point is this: Had I not polled the audience and just ASSUMED that the audience wanted to hear about dairy goats (remember our featured speaker) I not only would have been way out my comfort zone, would have bored the audience and most of them would have left early. As it was, we had stimulating discussion, great exchange of ideas and a successful program. What didn't happen was follow up evaluation.
In order to best serve our stakeholders, participants, companies and ourselves, we need to evaluate our programs, not only at the conclusion, but as follow up to evaluate behavior change as a result of the knowledge obtained during the course of the program. As educators, we can use the data gathered to measure our success as educators, the success of our teaching methods, the suitability of the material presented, benefits to our stakeholders, outcomes to use to gain more stakeholders (building reputation).
Who Should Conduct a Program Evaluation?
You can assemble many types of evaluation teams. Three possible options include:
1. Hiring an outside evaluator 2. Using an in house evaluation team supported by an outside consultant and program staff 3. Using an in house evaluation team supported by program staff
Hiring an outside evaluator. Many organizations typically do not have a research and evaluation staff and will probably need to hire an outside evaluator. This person would be supported by in house staff and would serve as a team leader. The evaluator could come from a research institute or a consulting firm.
Using an in house evaluation team supported by an outside consultant and program staff. If you feel that you have sufficient staff resources to implement the evaluation but need assistance with the technical aspects, you may want to hire an outside consultant. In this situation, an in house evaluator would serve as the team leader and be supported by both program staff end the outside consultant.
A consultant could support the evaluation by developing the evaluation design, conducting data analyses, and selecting or developing questionnaires. This person can also help you develop the evaluation plan and performance indicators.
Using an in house evaluation team supported by program staff. If resources are available within your situation, you could recruit these individuals to serve as evaluation team members.
Techniques
What are some fundamental techniques that program planners can use to collect data to evaluate programs? How should evaluation findings be reported?
Cafferella (2001) lists eleven elements of a systematic evaluation system:
1. Secure support for the evaluation from those who have a stake in the results of the evaluation. 2. Identify the individuals who plan and oversee the evaluation 3. Define precisely the purpose of the evaluation and how the results are to be used. 4. Specify what is judged and formulate the evaluation questions. 5. Determine who supplies the needed evidence and/or if some of that data are already available. 6. Delineate the evaluation approach. 7. Choose which data collection techniques to use, when the data are to be collected, and/or how the existing data can be put into usable forms. 8. Indicate the analysis procedures. 9. Stipulate the criteria to use in making judgments about the program or what process to apply in determining the criteria. 10. Determine the specific time line, the budget, and other necessary resources. 11. Monitor and complete the evaluation, make judgments about the value and worth of the program, and think through ways the evaluation data can effectively be used.
Techniques for collecting evaluation data, as mentioned by Cafferella (2001) include:
1. Observations 2. Interviews 3. Written Questionnaires 4. Tests 5. Product Reviews 6. Performance Reviews 7. Organizational and Community Records and Documents 8. Portfolios 9. Cost-Benefit Analysis 10. Focus Groups 11. Self-Assessment
When gathering and preparing evaluation information for reports and presentations, it is important to focus on the participants' learning, program operations, organizational issues, and societal issues. (Cafferella, 2001).
Cafferella (2001) also lists examples and descriptions of formats for reports:
1. Formal written reports 2. Executive summary of formal written report 3. Series of short written, oral, and/or pictorial reports 4. Briefings sessions 5. Draft report 6. Journalistic-style report 7. Media presentation 8. Case study report 9. Product display 10. Poster or display board 11. Oral report 12. Informational brochure 13. Electronic communication systems
Evaluation Use
How can evaluation be used to learn about your program outcomes?
The Need, The Response, The Results. This is the format we have used when reporting Extension program outcomes for stakeholders (Annual Reports, Legislative informational sheets, grant applications).
We, the educators use the community, local organizations and our clients to determine what the Needs are (we called it 'Scanning the Environment'). We then would develop a program around those needs (as long as it fit our mission statement) to provide the tools necessary for the anticipated changes needed (the Response). We then would use some form of evaluation to measure the Response (or impact) of the program on the lives of those who participated. These impacts result in part of our program outcomes, the changes made by the participants. Sometimes those outcomes were not the ones we expected and that's OK, as long as change can be observed or measured it still works for you the educator as well as your stakeholders.
The evaluation, if properly managed, will point to the actual changes made based upon the program. In a program on pasture management, participants may not actually alter the way they graze their animals (usually due to space and animal units), but they have a clearer understanding of what should be done and how. That is still a program outcome. Program Outcomes have a variety of usefulness. Depending upon your program objectives, what your stakeholders need, and what your mission statement is, your program outcomes should be useful tools, even if they measure your success as an educator planning programs for the adult learner.
Effective Communication Skills
I found that is a way for leaders to encourage learning among their employees. The five practices that Cheryl Zurawski, 2004 stated as the best practices to encourage learning in the office. 1. Use Questioning 2. Build partnerships 3. Shared experiences are case studies 4. Establish a clear and common purpose 5. Make time to talk. I think that these are good things to use in order to communicate in the office and if leaders in the workplace aren’t willing to incorporate these communication skills into a daily routine to get to know the employees then I don’t think that it would be a very fun work environment. I think everyone should understand these basic ideas so you can communicate with your leader and become a better employee.
Organizational Learning Model
Since learning about the organizational learning model there are many aspects that are very beneficial to learning. Things in the organizational learning model the areas that are covered such as collaborative decision-making processes, interdisciplinary work teams, global vision, a flattened organizational structure, and more opportunities for learning. Shahrzad Mojab and Rachel Gorman from the University of Toronto looked at this article on women and consciousness in the “Learning Organization”: Emancipation or Exploitation? I think that these are good concepts to take away from this article because you use them in an everyday job. I think that Adult learners will use these concepts all the time especially with their job.
Evaluation Resources
- Evaluation Wiki [[1]]
- Penn State Evaluation Fact Sheets [[2]]
- Basic Guide to Program Evaluation-this guide will help the beginning evaluator to set up an evaluation study. [[3]]
- Introduction to Evaluation-this is some basic information about evaluation. [[4]]
- Evaluation Learning model-this model breaks down the parts to a evaluation model to help you understand the areas.
[[5]]
- The Good, The Bad and The Ugly Evaluation-this site has evaluation criteria when using on-line sources. [[6]]
- Planning an Effective Program Evaluation:The Key to a Successful Program [[7]]
References
Cafferella, R. S. (2001). Planning programs for adult learners: A practical guide for educators, trainers, and staff developers 2 ed.). San Francisco: Jossey-Bass, Inc.
Dwyer, R. (2004). Employee development using adult education principles. Industrial and Commercial Training. 36(2/3), 79.
Hurley, R. (2002). Putting people back into organizational learning. Journal of Business and Industrial Marketing, 17(4), 270-281.
Mojab, S., Gorman, R. (2003). Women and consciousness in the learning organization: Emancipation or exploitation? Adult Education Quarterly. August. 53(4), 228-241.
Rogers, A. (2006). Escaping the slums or changing the slums? Lifelong learning and social transformation. International Journal of Lifelong Education. March-April. 25(2), 125-137.
