CEDTAP Forum 2000
Proceedings: Evaluating CED

CONTEXT

Sherri Torjman, chair of the session and Vice-President of the Caledon Institute of Social Policy, began by giving an overview of the contextual issues surrounding evaluation. She explained that we are currently living in a climate of accountability, which is the result of many different forces:

• Economic: the dominance of the market ideology in the post Cold War world has brought pressures to privatize or incorporate private sector practices such as results (bottom-line) management.
• Political: the failure of constitutional negotiations in the late 1980s and early 1990s led to non-constitutional processes; the Social Union Framework Agreement has built-in accountability provisions
• Social: local funders (private and community) face increased demands for funds as a result of more complex social problems and government cut-backs.
• International thinking: even the international concept of sustainable development has built-in notions of targets and indicators.

Ms. Torjman did stress that accountability is very important. The problem, however, is that CED organizations may be under pressure to conform to standard or traditional evaluation guidelines or audit requirements that may not fit. This lack of fit is due partly to the nature of CED itself which seeks to meet both economic and social objectives. This is what gives the field its unusual character and strength, but the social objectives are at risk of being lost. The challenge, then, is how to deal with this disjuncture:

• We need to recognize this challenge and engage in discussions with funders (government, private and community) about this tensions and indicate our interest in working to develop instruments that reflect more accurately what we are trying to achieve.
• Set reasonable targets and don't be pressured into unrealistic goals; otherwise, we run the risk of losing sight of our key principles of working with marginalized individuals, groups and communities.
• Develop new indicators that are appropriate to the work, that capture both outcomes and process.
• View the evaluation process as an opportunity for continual learning; it should not be a pass/fail exercise but more a question of what we have learned. This will help us move continually toward more intelligent, more informed and more strategic practice. We will continually raise the bar.

PARTICIPATORY EVALUATION OF THE WESTERN VALLEY DEVELOPMENT AUTHORITY

Approach to the Evaluation:

• Participatory - process as important as the products.
• Stakeholders involved in: defining goals of the evaluation; identifying indicators to assess; identifying methodologies to be used and establishing the workplan; and participating in research activities.

Why Stakeholder Participation?

• Capacity-building in the community.
• Establishing an "evaluation culture".
• Ensure appropriate context and interpretation of findings.
• Understanding of the steps and trust in the findings leads to greater commitment to implement recommendations.

Evaluation Framework:

Be as specific as possible:

• Based on the goals of the evaluation, draft an outline of the final report.
• Develop an evaluation framework to identify the following: what questions/issues need to be addressed; what is the appropriate indicator; and what is the source of this indicator and how will this information be gathered.

Be flexible:

• Provide the best advice possible, but allow decisions to be made by the community partner.
• Be willing to try new approaches.
• Be willing to move to "Plan B".
• Strive for perfection, but don't hold out for it.

Presenters:

Janet Larkman and Micheal Comeau, Western Valley Development Authority; Donna Crozier, Praxis Research and Consulting;

OPPORTUNITIES 2000:  EVALUATION AS COLLABORATIVE, EXPERIENTIAL LEARNING

OP2000: Vision, Goals, Strategies, Structure

Vision

We envision the people of Waterloo Region who are living in poverty working with people from all sectors of the community to create opportunities which will reduce the region’s poverty rate to the lowest in Canada and in so doing generate learnings and set an example for all Canadians.

 Goals

1. Help 2,000 families move out of poverty by December 31, 2000 and by doing so help Waterloo Region achieve the lowest poverty rate in the country.
2. Identify and implement the most practical methods for creating employment and income opportunities that can lead people out of poverty.
3. Share the project’s lessons and successes with other communities across Canada.

Strategies

1. Leadership: rely on community leadership to make poverty reduction an acknowledged public priority across Waterloo Region.
2. Research: conduct original research into poverty and poverty-reduction strategies and disseminate these findings across Canada.
3. Projects: assist local organizations, businesses and governments in the development of projects that create new income or employment opportunities for people living in poverty.

Revised Evaluation Framework:

Overall Aims

1. Chronicle the OP2000 experience.
2. Document the diverse array of outcomes it achieved.
3. Identify the strengths and weaknesses of the strategies employed.
4. Determine the lessons gained for future efforts at poverty reduction.

Dimensions

1. The multiple levels at which OP2000 is operating.
2. The evolutionary nature of OP2000 and its various projects.
3. Diverse aspects of change (personal and institutional; quantitative and qualitative; process and outcomes).

Outcome Levels

1. Individuals: Income and non-income changes occurring in the lives of project participants and their households.
2. Organizations: Changes in the way partner organizations perceive their roles in relation to poverty reduction work and in their capacity to play these roles.
3. Community: Changes in the commitment of the wider community to reduce poverty and in its ability to act in a concerted fashion to achieve this goal.

Methods:

1. Individuals

• Household survey (basic demographic data and key data for assessing change in relation to the Low Income Cut-Off).
• Change in level of income.
• Change in composition of income (government assistance vs. employment).
• Change in depth of poverty (extent to which the household falls below the Low Income Cut-Off).
• Participant profiles in brief (brief accounts of gains made and difficulties encountered by project participants; collected quarterly).
• Hope/confidence.
• Belonging.
• Security.
• Personal support networks.
• Access to services.
• Skills.
• Income.
• Employment.
• Policy issues.
• In-depth interviews (fine-grained accounts of how project participants—approximately 30—came to be in poverty and where OP2000 fits in their efforts to move out of poverty).
• Life history (including: intergenerational factors; changes in circumstances over time).
• Interdependent and layered influences .
• Changes in social context.
• Key moments/people/events.

2. Organizations

• Learning diary (compilation of original project proposal, quarterly progress reports and quarterly self-assessments).
• Assessment of organizational capacity.
• Development of innovative poverty reduction program/strategy.
• Identification of new target groups.
• Resource acquisition.
• Lessons learned during implementation.
• Project interviews/focus groups.
• Changes in organizational policies.
• Changes in understanding of poverty.
• Acquisition of skills and assets.
• Inclusion of people in poverty in decision-making processes.
• Formation of new partnerships.
• Enhanced communication with other groups/organizations/sectors.
• Ability to influence public policy.
• Willingness/capacity to pursue poverty reduction activities in the future.

3. Community

• Key informant interviews/OP2000 staff learning diaries.
• Awareness and understanding of poverty problem.
• Commitment to poverty reduction.
• Shared vision/long term plan.
• Mobilization of resources.
• Collaboration among diverse stakeholders.
• Creation of new structures and/or institutional processes.
• Changes in public policy.

Evaluation as Collaborative, Experiential Learning:  10 Notes

1. Focus on learning, not grading.
2. Affirm the nature of the project as a collaborative learning process and the participants as learners.
3. Focus more on underlying theories and less on outcomes.
4. Create opportunities for participants to reflect in some depth.
5. Address the full range of activities in which the project is engaged.
6. Feed back in a clear way what people express in a less clear way.
7. Boil down complex issues into manageable challenges.
8. Present a balanced picture including the diverse and sometimes divergent perspectives of the participants.
9. Pose the issues as challenges for the project to address.
10. Streamline the process and provide frequent input.

Presenter

Eric Leviten, Caledon Institute of Social Policy

LEARNING AND ASSESSMENT FRAMEWORK

Challenges Facing Organizations in Beginning to Evaluate Their Projects:

• Complexity of evaluation.
• Resistance to "measurement" and "results".
• Lack of knowledge/expertise.
• Time and expense.
• Lack of ownership.
• Control of learning agenda.

Definition of a Learning and Assessment Framework:

A learning and assessment framework is a planning tool that assists organizations:

• To clarify the purpose of projects and/or programs.
• To clarify strategies and work plans for meeting those priorities.
• To state the expected long and short-term results of activities.

Rationale:

• Focuses resources and energy.
• Builds common priorities.
• Gets you organized for action.
• Supports you to document your results.
• Strengthens your impact.
• Identifies what you want to learn.
• Clarifies terms of accountability.
• Builds a foundation for assessing projects.

Why Pursue this Approach?

• To ensure ownership of the learning process.
• To demonstrate the impact of your work.
• To set clear priorities.
• To show that your money is well spent.
• To support sharing of learning.
• To raise the profile of CED and the initiative.
• To promote better practice.

Advantages of the Approach:

• Creates ownership of learning.
• Increases confidence and competence in results tracking.
• Increases accountability.
• Develops more realistic statements of results.
• Strengthens planning.
• The process has supported teambuilding within organizations and and partnership development between organizations.
• Reduces distrust/competition.
• Transferability of the tool - thinking and planning process already being applied elsewhere by organizations.

Presenters

Janet Murray and Mary Ferguson, Eko Nomos

Resources

Are Outcomes the Best Outcomes? by Sherri Torjman, Caledon Institute of Social Policy
Evaluation Report of the Western Valley Development Authority
"A Framework for Evaluating Multidimensional CED", by Eric Leviten, Making Waves, Vol. 11 No. 2