By Ruth Stewart and Laurenz Langer
When I (Ruth) first learnt about systematic reviews it was in an undergraduate lecture on research methods. We were learning about randomised controlled trial methodology and those who work to synthesise the lessons within them. My lecturer referred to systematic reviews as a new methodology (this was 20 years ago), and commented that those who conducted them were a little pedantic. Little did I know that 20 years later I would be leading a research centre with a track record for rigorous and relevant evidence synthesis. Our work on evidence synthesis is led by my dedicated and talented colleague, Laurenz Langer. The portfolio of work described below is largely credit to a collective effort of an impressive team of research staff, which since 2012 has involved 23 African researchers conducting evidence synthesis.
Over the last few months we have been asked a number of times for more information on our track record on producing systematic reviews and evidence maps. In pulling together the lists of our work, we realised there is more to say than just listing our projects. In truth there is more to say about how we at the Africa Centre for Evidence (ACE) aim to produce systematic reviews and evidence maps.
ACE’s evidence synthesis is about supporting development and transformation in three ways. First, it is about making sure that scarce public resources in low- and middle-income countries (LMICs) are used for the most benefit. This includes supporting decision-makers with rigorous and relevant bodies of knowledge to design and implement the most effective social policies and programmes. Evidence synthesis contributes this knowledge of what works. In addition, we also need to make the most of what we already know, to avoid duplication of research and thus enhance efficacy in research funding. Second, it is about enhancing transparency and accountability in decision-making processes. Policies and programmes openly based on bodies of research evidence can support the public’s trust in the policy development process. Third, it is about empowering those with little access to the policy arena and who have most to lose from ineffective policies. For the most marginalised communities in LMICs, the opportunity cost of failed policies (e.g., lack of electricity, poor public schools) is much larger than for more affluent constituencies. In addition, these groups are often located away from the centres of power (e.g., in rural areas) or historically have not been included in policy-making (e.g., for reasons of race and gender). Evidence synthesis brings together the full available body of knowledge, not just the knowledge and evidence of groups well positioned in the policy process. It thereby can give less powerful actors a voice in this process and highlight knowledge and data gaps relevant for these groups (e.g., a lack of gender-disaggregated data).
This has a number of implications for our work:
- We aim to work closely with decision-makers whenever possible. At the very least our reviews are informed by advisory groups that include decision-makers, but this is the very minimum for ACE. We prefer where ever possible to co-produce systematic reviews and evidence maps with decision-makers, usually those in government. To date this has included working with the South African departments for Planning, Monitoring and Evaluation, Environmental Affairs, Science & Technology, Human Settlements, Rural Development and Land Reform, and Basic Education; and a range of international bodies and governments, such as the European Union, the UK’s Department for International Development, the Canadian International Development Agency, and the International Fund for Agricultural Development.
- We are pragmatic about timelines. A systematic review or an evidence map that misses a policy moment is limited in its value to inform that decision. If colleagues ask us for a systematic overview of the evidence base tomorrow, they will get the best we can do by tomorrow. We prefer a few weeks, and months would be nice, but we will do all we can to get the evidence to the table on time.
- Linked to 1 and 2 above, we don’t always do full systematic reviews, or even full evidence maps. Our aim is to be responsive. This often means that there is not always a formal protocol, or even a full report. This is not something we are entirely happy with, and we are working to develop some ‘process documents’ to allow others to learn about and review our methods, despite our inability to produce protocols for each and every piece of work.
- We work across social policy areas from housing programmes to technology use in education, and building state capability. We are in theory happy to work across all the chapters of South Africa’s National Development Plan, and all the 17 of the Sustainable Development Goals. For a full list of policy areas, see below.
- As 4 above suggests, we work across sectors. We are happy to provide methodological input to reviews and maps on subjects that we may not know a lot about. We employ researchers from a range of disciplines and backgrounds with the ability to learn fast about new areas. And perhaps most significantly we always partner with others who can help to fill any gaps in our own skills and knowledge. Over the last five years, this has, for example, included: UCL EPPI-Centre, 3ie, McMaster Health Forum, UK What Works Centres, South Africa’s Cochrane Centre, South Africa’s Medical Research Centre, Witwatersrand’s Centre for Health Policy, the Centre for Evidence-based Health Care at Stellenbosch University, CLEAR-AA, the Centre for Rapid Evidence Synthesis at Makerere University, AFIDEP, ZeipNet, VakaYiko, the Centre for Industrial Development at the University of Johannesburg, Department of Education at the University of Oxford, the EU Commission, the Campbell Collaboration, the Collaboration for Environmental Evidence, and many more.
- We are constantly developing our methods. We are trained and experienced in the most thorough systematic review methods that our ‘proper geeky’ colleagues at Cochrane, Campbell and CEE are specialists in. Across our team we have expert knowledge in different types of reviews and synthesis including effectiveness reviews with statistical meta-analysis, qualitative evidence synthesis including thematic synthesis, and anything in between to configure and aggregate bodies of knowledge (e.g., qualitative comparative analysis, conceptual synthesis, umbrella reviews). We produce full systematic reviews registered with review umbrella organisations as much as tailored products involving rapid and responsive synthesis and evidence mapping approaches. In general, our starting point is to explore mixed-methods reviews and to adapt these methods to the specific needs and questions from our partners.
And we are also experienced at assessing the relative risks of adapting the ‘full’ methodology to meet the timelines and specifications of a policy priority. We are constantly innovating, learning and adapting from a position of expert knowledge. (Our aim is to never do these pragmatic adaptations in ignorance.) ACE team members have played a role in the innovation and development of evidence synthesis as a research methodology in international development and in environmental management (see last list below for relevant publications).
- We have developed new technology for presenting our findings to various audiences. To date this primarily focusses on visualisation from evidence maps and evidence databases (see here and here for examples). We have also adapted low-cost collaboration tools, such as Google Sheets, for evidence synthesis more suitable for contexts in which our partners don’t have access to formal review software. We also support others in the development of their own evidence synthesis technologies, such as DPME’s evidence mapping software.
- Last but not least, we are committed in all of our evidence synthesis to learning from and contributing to ACE’s other key portfolios: evidence communities and evidence capacities, as well as advancing the theory and methods of evidence use. In practical terms this includes running training in evidence synthesis, and focussing on relationship building through new and existing networks and communities of practice, in the strong belief that only together can we make evidence-informed decision-making a reality in our region.
And finally, to answer the question of what systematic reviews and maps we have produced as a team, here is the full list (as of Feb 2019):
Systematic reviews (registered with review umbrella bodies)
- A systematic review of urban agriculture in LMICs (CEE)
- A systematic review of smallholder farming interventions in Africa (Campbell)
- A systematic review of certification schemes in LMCIs (Campbell)
- A systematic review of agricultural subsidies in LMICs (Campbell)
- A systematic review of payment for ecosystem services in LMICs (Campbell)
- A systematic review of Interventions to improve the labour market situation of adults with physical and/or sensory disabilities in LMICs (Campbell)
- A systematic review of women in wage labour in LMICs (EPPI)
- A systematic review of what works to support evidence use (EPPI)
- A systematic review of microfinance interventions in Sub-Saharan Africa (EPPI)
- A systematic review of microfinance interventions in LMICs (EPPI)
- A systematic review of mobile learning for development (EPPI)
- A systematic review on working with non-state providers in post-conflict and fragile states in primary healthcare service delivery (EPPI)
- A systematic review on teaching critical appraisal skills in healthcare settings (Cochrane)
- A systematic review of ICT in education (Campbell protocol)
Other evidence synthesis
- A scoping review of educational models for mHealth and CHWs in LMICs
- A scoping review of the models and methodologies for rapid evidence synthesis services
- A rapid evidence assessment on protected areas and poverty reduction in Africa
- A rapid evidence assessment on marine conversation and poverty reduction in Africa
- A rapid evidence assessment on HIV/SRHR programming with and for transgender people in Sub-Saharan Africa
- A rapid evidence assessment on methods used in ecosystem services and poverty alleviation research
- A rapid evidence assessment on decision-tools for ecosystem services and poverty alleviation
- A response evidence synthesis on the state of land degradation in KwaZulu-Natal
- A responsive evidence synthesis on land reform in South Africa
- An overview of digital models for edtech in the European Union
- An evidence database for agricultural research for IFAD
- A systematic map of smallholder farming in Africa
- A systematic map of sport for development interventions in Africa
- A systematic map of studies of patients’ and clinicians’ research priorities.
- A policy-relevant evidence map on human settlements and housing in South Africa (co-produced with DPME)
- A policy-relevant evidence map on research on the concept of the Developmental State (co-produced with DPME)
- A policy-relevant evidence map of early-grade mathematics teaching and learning in South Africa (co-produced with DPME)
- A policy-relevant evidence map of evidence on the national spatial development framework in South Africa (co-produced with DPME)
- A policy-relevant evidence map of land reform in South Africa (co-produced with DPME)
- A policy-relevance evidence map on eco-system services for poverty alleviation in Africa (co-produced with DEA)
- A policy-relevant evidence map of environmental research evidence in South Africa (co-produced with DEA)
- A policy-relevant evidence map on innovation and inclusive industrialisation in South Africa (co-produced with DST)
- An evidence map on violence against women & children in South Africa
- An evidence map on ICT in HIV/SRHR programming with and for transgender people in Sub-Saharan Africa
- An evidence map on the prevalence of abuse among children living with disability in East Africa
- An evidence map on the use of mobile technologies to support the training of CHWs in LMICs
- An evidence map of what works to support women’s wage labour participation in LMICs
- An scoping evidence map of student support initiatives in higher education teaching and learning
Some of our publications on evidence synthesis methods and evidence synthesis thinking:
- Langer L, Erasmus Y, Tannous N & Stewart R (2017) How stakeholder engagement has led us to reconsider definitions of rigour in systematic reviews. Environmental Evidence 6(20). DOI 10.1186/s13750-017-0098-7.
- Stewart R, Langer L & Dayal H (2017) Titles, tensions and terminology within evidence-informed decision-making. Research for All 1(2): 252-264. https://doi.org/10.18546/RFA.01.2.03.
- Gough D, Stewart R &Tripney J (2017) Using research findings. In Gough D, Oliver S & Thomas J (2017) An Introduction to Systematic Reviews. London: Sage Publications: 279-296. ISBN: 9781473929432.
- Africa Evidence Network (2017) Capacity in Africa: The results of a survey on support for and production of evidence maps and evidence syntheses, including systematic reviews. AEN: Johannesburg, SA.
- Haddaway NR, Kohl C, Rebelo da Silva N, Schiemann J, Spök A, Stewart R, Sweet JB & Wilhelm R (2017) A framework for stakeholder engagement during systematic reviews and maps in environmental management. Environmental Evidence 6(11). DOI 10.1186/s13750-017-0089-8.
- Dayal H & Langer L (2016) Policy-relevant evidence map: A departmental guidance note. DPME: Pretoria, SA.
- Da Silva NR, Zaranyika H, Langer L, Muchiri E & Stewart R (2016) Making the most of what we already know: A three-stage approach to systematic reviewing. Evaluation Review. DOI: 10.1177/0193841X16666363.
- Stewart R (2014) Changing the world one systematic review at a time: A new methodology for making a difference. Development Southern Africa 31(4): 581-590.
Langer L, Stewart R (2014) What have we learned from the application of systematic review methodology in international development? – A thematic overview. Journal of Development Effectiveness 6(3): 236-248. Open access: http://www.tandfonline.com/doi/abs/10.1080/19439342.2014.919013#.VKgfylvlfzI
 Not counting the government departments listed above.