Evaluating public policy grantmaking is different than evaluating traditional social service programs. Because policymaking is a slow process influenced by many factors, it is often difficult to determine what effect any particular funder or grantee has on a policy outcome. Because of this, evaluation frameworks seeking to identify causality often miss the mark when assessing public policy engagement.

Different types of public policy engagement require different methods of evaluation. Accurate evaluations of public policy grantmaking must take context into account. By identifying the incremental steps that lead to successful policy change, funders can determine appropriate performance measures.

Necessary factors to create policy change include:

• policymaker and media education

• broad based coalitions

• champions of the issue

• long-term commitment to the issue

• growth in civic participation

These factors are critical not only to a particular policy change but also for ensuring that policy victories are maintained and expanded for lasting social change. When a particular policy objective is not achieved, funders must remember that grantmaker support lays the groundwork for future victories. Evaluations that assess the means of achieving policy change reflect a more accurate picture of grantee and grantor performance than those that focus on policy outcomes.

Large policy wins cannot happen without small steps that pave the way. When assessing policy formation and change, evaluations cannot only measure the outcome; looking at process is key because policy change is often incremental and long-term. It is important for funders to encourage grantees to evaluate themselves through an ongoing assessment of their work to inform their decision-making and future efforts.

Three Levels of Public Policy Assessment

Generally, there are three levels for assessing public policy efforts:

• Institutional level (changes in policy enactment and implementation)

• Community level (social capital; how a community views an issue)

• Organizational level (organization's capacity to impact public policy)

Case Studies

In 2004, NCG commissioned a report on evaluating public policy grantmaking: Evaluating Philanthropic Support of Public Policy Advocacy: A Resource for Funders. The report details six case studies on assessing public policy grantmaking:

• The David and Lucile Packard Foundation

• The Woods Fund of Chicago

• The Rosenberg Foundation

• The Liberty Hill Foundation

• The San Francisco Foundation

• The PICO California Project

• Full report--Evaluating Philanthropic Support of Public Policy Advocacy: A Resource for Funders

The case studies highlight the following lessons learned:

• Combining quantitative and qualitative data strengthens evaluation efforts.

• When possible, indicators should be based in a strong theory of change or long-term strategy.

• Policy advocacy evaluations should focus more on improving practice than on improving impact (or on both). Evaluation should adopt a continuance improvement model which frames the process as "performance improvement" rather than "evaluation."

• Value narratives and storytelling in evaluation.

• Measuring collaboration is critical to advocacy assessment.

• Evaluations must take into account the dynamic policy environment.

• Evaluations should monitor the depth and sophistication of advocacy strategies.

• Grantees should be involved in evaluation design.

• Evaluations can be used to build grantee capacity.

• Discerning attribution is difficult but not impossible.

• Evaluations should include an assessment of the funder's role.

• Evaluations should measure policymaker perceptions.

• Evaluations must recognize the relationship and tension between producing policy results and constituency building.

• Evaluations must distinguish between membership and leadership development.

• Evaluations should include the broader community.

Resource Links

• Alliance for Justice's "Investing in Change: A Funder's Guide to Supporting Advocacy" has a chapter on evaluating and reporting advocacy. 
• Interview with Paul Brest, President, Hewlett Foundation, Alliance for Justice
• Making the Case, The Women's Funding Network
• Tips for Evaluating Advocacy" Alliance for Justice
• "Tips for Evaluating Advocacy—A Checklist for Grantees" Alliance for Justice

From http://www.ncg.org/toolkit/html/diggingdeeper/assessing/print_assessing.html