Steps

Steps in the Process

A good way to begin is to create a logic model that enables you to define what you hope to change, how you hope to accomplish the change(s) and how you will measure progress and results all along the way.

1. Create a UWSF Logic Model.

   A logic model can be defined as "a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve" (Kellogg Foundation, 2004, p.1). Logic models have many purposes, including:

   • Making your underlying beliefs explicit,
   • Challenging assumptions and strengthening logic,
   • Communicating the initiative's essential purpose to broad audiences, and
   • Making the case for more financial investment. (Kellogg Foundation, 2004).

We have provided a logic model template (Figure 1) and two sample logic models prepared by two local United Ways in their Strengthening Families work. The outcomes that you identified in Step 3 of the Planning stage are the foundation for your logic model. In essence, the logic model requires you to identify short-term, intermediate, and long-term outcomes, along with the resources, activities and strategies, and target audiences for your effort. Good logic models also incorporate strong thinking about how and why your approach will likely produce the intended outcomes. The development of a logic model is an excellent way to make sure the planning team has a shared understanding of what the approach will be; the logic model can also serve as a guide for how well you are implementing your approach (Planning step 6) and what aspects of the program you will want to evaluate (Planning step 7). The major elements of a logic model are the following (see the Resources section for outstanding logic model sources):

• Inputs. Most logic models reference "inputs", but what we really mean is resources that are necessary to make your program or initiative a reality. This can include funding, personnel time, equipment, necessary partnerships, and so on.
• Activities. The activities you list define what your project or initiative will do. In the case of UWSF, your activities should be closely aligned to the approaches you have selected. If training is a primary focus, then the activities should be training-related.
• Outputs. Outputs refers to two things: (1) products or deliverables that are created to accomplish your activities (this can include things like classes taught and curricula developed); and (2) participation and engagement, which refers to who is served, as well as how and why they are engaged. This should include your target audience, as well as some understanding of their motivation to be engaged.
• Outcomes. Outcomes are the changes that take place in your target population and should have been identified in Step 3 above. It is important that these outcomes be phrased in terms of how people change (not what we do for people). Outcomes can be broken down into short-term, intermediate, and long-term, with some logical understanding of what outcomes lead to longer-term outcomes. In United Way, this is often referred to as the "chain of outcomes". These outcomes include changes in people such as knowledge, attitudes, awareness, behaviors, and new practices.

Figure 1. Logic Model Template

Image

2. Create an Evaluation Plan.

   The evaluation plan should be developed in concert with the logic model. The logic model should include measurable outcomes, including both process questions (related to resources, program delivery, audience engagement, etc.) and outcome questions (Is the program achieving its intended outcomes?). As you create the logic model, you can also define how you will evaluate each element in the model. We recommend that you use the Wisconsin Cooperative Extension created "Planning a Program Evaluation" and the accompanying worksheet to guide the development of your evaluation plan. Based on this Wisconsin Cooperative Extension worksheet, the key steps to developing an evaluation plan are:

• Engaging Stakeholders: If you engaged the right people in your Community Planning Team, then you have already engaged the proper stakeholders; however, you may wish to revisit this issue in the context of evaluation to determine if any new stakeholders should be engaged.
• Focusing the Evaluation: At this point, you should clarify: (1) the purpose of the evaluation; (2) the intended users of the evaluation; (3) the evaluation questions; (4) the indicators that will help you answer your questions; and (5) the evaluation design you will use. You should also reconsider the measures and instruments identified in Step 3 of the Planning stage.
• Collecting the Information: You will need to decide how you will collect the data to answer your questions and also clarify how you will manage the information you are collecting.
• Analyzing and Interpreting the Information: In this portion of the plan, you will identify: (1) how the data will be analyzed; and (2) who will analyze the information.
• Using the Information: Because you identified intended users earlier, it should be easier to think about how you will use the information. You will decide: (1) who to share the information with; (2) how to share the information with various groups; and (3) how to use the information for reporting outcomes and improving the project.

With this evaluation plan in hand, you have a better chance to proceed with an evaluation approach that is both thoughtful and useful.

3. Learn, Modify, and Improve.

   Michael Patton says that, "In the simplest terms, evaluations are said to answer three questions: What? So what? and Now what?" These three questions describe much of what we can learn from evaluation:

   • What? "What" questions include: What are we seeing? What is happening? What do we now understand about our program or initiative?
   • So What? "So what?" is a classic evaluation question that urges us to get to compelling descriptions of outcomes and impacts. What is changing as a result of our efforts? Are we having an impact? Why do we care?
   • Now What? This is the action question. Now that we have learned something about our program, what do we do differently? How do we change and adapt our approach so we are more successful in the future?

Patton has also stated that this last set of questions can be the most difficult to answer:

"The hardest thing I find to teach is how to go from data to recommendations. When you are doing an evaluation, you are looking at what has gone on -- a history. But when you write recommendations, you are a futurist. Evaluations can help you make forecasts, but future decisions are not just a function of data. Making good, contextually grounded, politically savvy and do-able recommendations is a sophisticated skill."

While it is a difficult task, good planning can lay the foundation for better use of your evaluation findings.

Keys to Success

• Identify your intended users at the beginning of the process (as part of the evaluation plan) and make sure you collect information that will be useful to these intended users.
• Include project staff in this list of intended users and work hard to collect information that informs both outcomes and process so you can understand not only what outcomes have been achieved, but also gain some understanding of why or why not.
• When thinking about ways to share information with key audiences, "start with the end in mind" by developing an outline or basic structure for a presentation or report. Make sure the information you collect fits cleanly into your organization (and consider eliminating data collection for items that you do not plan to share with key audiences).