Review of Quality of Evidence Supporting CRP Claims of Contribution to CGIAR System Level Outcomes 2017-2021

Background:

During the period 2017-2021, CGIAR Research Programs (CRPs) submitted claims of contributions to System Level Outcome (SLO) indicator targets for 2022. These claims were supported by evidence, and in most cases, a high-quality study (e.g. peer-reviewed publication) was expected to be provided. A formal quality assurance process was conducted annually to ensure that supporting evidence substantiated the claims. In addition, in early 2023 the Portfolio Performance Unit commissioned an assessment of how well CRP claims of contributions to 2022 SLO targets aligned with the specific indicator and could be aggregated across the CGIAR. This study aims to fill that gap. Both of those exercises checked to see whether there was evidence submitted in support of the claim and that the evidence spoke explicitly to the claim made. However, neither the quality assurance process nor the commissioned assessment reviewed the rigor of the methodologies used to produce the evidence. This study aims to fill that gap.

Requirements

Task and deliverables:

The main task is to review evidence submitted to support claims of CGIAR’s contribution to the 2022 SLO targets. Sub-tasks include:

a) developing a methodology for assessing the evidence,

b) assessing the evidence according to the methodology (e.g. providing an assessment of the percentage of papers that meet various quality criteria),

c) summarizing the findings including the identification of the most common weaknesses and

d) making recommendations for improvement.

The deliverable will be a single report that covers all the sub-tasks a) through d).

Evidence:

The consultant will work with a dataset of coupled claims and links to evidence compiled from the submissions of the CRPs.

There are about 120 such cases, spanning about ten different targets. However, the bulk of the submissions were related to the use of an innovation or yield effects resulting from the use of an innovation. Of the 120 cases, more than half were for two of the targets – use of innovations and contributions to yield increases. For those two targets, a sample of studies can be drawn and for our purposes, a final sample of about 80 papers to review should be sufficient.

Methodology Development Process:

The consultant will develop a methodology for assessing the evidence submitted. The process for developing the methodology will involve the following steps:

• The consultant will propose and justify a methodology for assessing the quality of evidence used in CGIAR impact assessment studies.
• Pilot Testing: To ensure the appropriateness and feasibility of the methodology, the consultant may conduct a pilot study on a subset of the dataset. This will allow for refinement of the methodology, identification of any challenges or limitations, and validation of the proposed approach.
• The consultant will present the proposed methodology to the Portfolio Performance Unit for review and feedback before proceeding with the analysis.

Qualifications:

We seek an individual or team led by an individual with a strong record of publication on related themes such as the adoption of innovations by farmers, ex-post impact assessments of innovations, systematic reviews, and/or rigorous causal impact evaluations.

Timeline:

We anticipate the total number of person-days for this task is about 15 days (including 2 to develop the methodology, 10 for analysis and 3 for write-up). The deadline for submission of the final report would be 45 days after the signing of the contract.

Benefits

CGIAR is committed to fair, safe, and inclusive workplaces. We believe that diversity powers our innovation, contributes to excellence, and is critical for our mission. We offer a multi-cultural, multi-color, multi-generational, and multi-disciplinary, collegial working environment, with a competitive salary, excellent benefits, and flexible working arrangements. We are consciously creating an inclusive organization that reflects our global character and our commitment to gender equity. We, therefore, encourage applicants from all cultures, races, ethnicities, religions, sexes, national or regional origins, ages, disability status, sexual orientations, and gender identities.