Ongoing PDP Evaluations
• EVI (formerly EMVI)
• 2005: Mid Term Review, 2005: KPMG final report May05.pdf
• Commissioned by: EMVI
• 2012: Mid Term Strategic Review, 2012: EVI_External review_Report_121213.pdf
• April 2003: Evaluation of the International AIDS Vaccine Initiative: 2003 INDEP ASSESSMENT_file.pdf
• Commissioned by: IAVI, World Bank
• February 2009: Evaluation of the International AIDS Vaccine Initiative: 2003-2007: Evaluation of IAVI 2003-2007_Final Report.pdf
• Commissioned by: IAVI, World Bank
• Related Materials:
• IAVI Management Response to the Evaluation. March 2009: IAVI Response to 2003-7 evaluation_FINAL.pdf
• June 2008: International Partnership for Microbicides Evaluation Report
• Commissioned by: Evaluation Management Group of IPM Donors.
• Funded by: Irish Aid, BMGF, CIDA, DGIS, Norad, DFID
• Related Materials:
• IPM Management Response to Donor Evaluation at Year 5. October 2008: IPM Management Response to Evaluation 2008.pdf
• Funders Response to the IPM Evaluation. January 2009: IPM evaluation- funders response - final 14 Jan.doc
• May 2005: Independent Review of Medicines for Malaria Venture
• Commissioned by: DFID, Wellcome Trust, World Bank, Swiss Agency for Development and Cooperation, BMGF, DGIS
• June 2007: GPR_medicines_malaria.pdf
• Commissioned by: World Bank
• Dutch External Evaluation of PDPs (IAVI, IPM, Aeras, TB Alliance, FIND, MMV, DNDi, EMVI)
• November 2009: DGIS_PDP_Evaluation_-_Final _25 Nov.pdf
• Commissioned by Netherlands Ministry of Foreign Affairs (DGIS)
• Irish Aid Review of Support to Product Development Partnerships
• June 2011: Irish Aid Review of PDPs FINAL 290611.pdf
• PATH Vaccine Product Development Assessment:
• Summary of Findings.The Boston Consulting Group (BCG) July 2010. PATH Vx DEV EVALUATION-BCG-15Dec10-vfinal-2.pdf
• 2011. Assessment conducted by BCG and commissioned by BMGF & Netherlands Ministry of Foreign Affairs (DGIS). Report not in the public domain.
Lessons learnt from PDP joint evaluations conducted to date
• MMV: May 2005 Joint Evaluation: MMV evaluation process Jan 14.pdf
• IPM: February 2009 Joint Evaluation: IPM evaluation - review of the process- Oct 17.doc
Gates- PFG Project on Evaluations conducted by FSG
Toward a New Approach to Product Development Partnership Performance Measurement. FSG Social Impact Advisors. June 2007. TowardaNewApproachtoProductDevelopmentPartnership(PDP)PerformanceMeasurement-1.pdf
Key findings from the analysis include:
• Donors vary greatly in the scope and rigor of performance oversight;
• The current state of practice in PDP performance measurement lacks structure and comprehensiveness;
• All PDPs measure common areas such as governance, people, finance, portfolio and R&D project management, but they differ considerably in the:
• Manner in which they measure individual activities or goals;
• Measurement priorities reported by management;
• Sharpness with which they articulate their metrics.
• It is possible to classify PDP performance metrics in a way that makes sense for PDPs and donors.
After assessing the state of performance measurement among PDPs, this paper presents a new performance measurement framework for PDPs. The framework will help donors and PDPs link strategic planning to performance measurement and more clearly communicate about their progress.
The framework offers a more comprehensive set of issues than the conventional array of R&D performance metrics. Four Areas of Performance – R&D to Commercialization, Organizational Strength, Enabling Environment, and Health Impact – reflect the challenging reality of PDPs’ efforts to bring new technologies to bear.
The framework is an important – but not final – step towards improving the state of performance measurement among PDPs and donors. Interviews revealed needed changes in current donor evaluation processes so that they become more consistent, predictable, less burdensome and valuable to both donors and PDPs. FSG recommends that a new “performance measurement partnership” be established between donors and PDPs to improve the value of donor-sponsored evaluations and internal management reviews by PDPs. Such a new performance measurement relationship would seek to:
− Build on pre-investment assessments and periodic business planning efforts;
− Complement internal staff and board performance monitoring processes;
− Enhance insight generation by providing appropriate support for external information gathering and assessments (e.g. R&D benchmarking, market assessments, health impact assessments);
− Go beyond validation of PDP decisions and investments to jointly identifying constraints to success and sharing responsibility where appropriate for achieving and measuring progress towards key objectives of PDPs.
Minutes from discussions related to the FSG report.
• PDP Meeting - July 2007: Summary of PDP meeting on Perf Measure 26 Jul 2007.pdf
• Joint PDP and Funder Meeting - October 2007: PDP Funders WG Oct 2 Minutes.pdf
FasterCures Philanthropy Advisory Service - Organizational reports
The FasterCures metrics measure an organization’s operational process as well as its contribution to the field of disease research. Metrics are aggregated into four major categories – accountability, collaboration, research effectiveness, and resource building. Assessment results are intended to provide guidance to donors about performance relative to an organization’s mission, and do not represent investment recommendations.
For more information see: http://www.FasterCures.org/Programs/PAS.php
For access to one of the following reports visit the FasterCures Philanthropy Advisory Service website
• TB Alliance
Other related evaluations/ papers
• IVI Evaluation
• The relevance and future role of the International Vaccine Institute (IVI) 2000-2006. Commissioned by: SIDA. September 2007
• EDCTP Internal Assessment
• Internal Assessment of the 2003-2009 EDCTP Programme. Commissioned by EDCTP. October 2009
• Malaria Vaccine Decision Making Framework
• Survey Evaluation of the Malaria Vaccine Decision Making Framework Process. Commissioned by USAID. December 2008:
• Case Studies: Results from Grants from the Bill & Melinda Gates Foundation (BMGF)
• Developing a Safe, Inexpensive Cure for ‘Black Fever’. A “What We’re Learning Report” from the Bill & Melinda Gates Foundation. June 2006
• Supporting Development of a Vaccine for Malaria, the Deadliest Disease Among Africa’s Children. A “What We’re Learning Report” from the Bill & Melinda Gates Foundation. August 2006
Articles on assessing/ evaluating PDPs
• Demonstrating value: Performance metrics for health product development public-private partnerships. Marc Pfitzer. In Combating diseases associated with poverty: Financing strategies for product development and the potential role of public-private partnerships. 2004.
• Areas for Future Attention. Roy Widdus and Katherine White. In Combating diseases associated with poverty: Financing strategies for product development and the potential role of public-private partnerships. 2004
Other articles on evaluations
• A framework for mandatory impact evaluation to ensure well informed public policy decisions. AD Oxman et al. The Lancet (Viewpoint). 375 (9712). January 30.
• Measuring impact in the Millennium Development Goal era and beyond: a new approach to large-scale effectiveness evaluations. Victora CG et al. The Lancet, Volume 377, Issue 9759, Pages 85 - 95, 1 January 2011
• Mathematical models in the evaluation of health programmes. Garnett GP et al. The Lancet, Early Online Publication, 11 April 2011
• The art of evaluating the impact of medical science. L Allen. Bull World Health Organ. 2010; 88:4