By Tanya Mdlalose and Metoh Azunui with support from Promise Nduku
When one reaches a milestone in a journey, it is important to reflect on the path taken so far, the joys, trials, and tribulations faced. Our work on the South African environmental evidence map has reached an important milestone, giving us a moment to pause and reflect. In this blog post, Tanya and Metoh reflect on their experiences as research assistants conducting searching and screening for the first time.
We have completed screening for inclusion on title and abstract of more than 150 000 citations, leaving us with about 7 000 citations to screen on full text for inclusion in an evidence map on all environmental research focused on South Africa, aptly referred to here as our mega map.
Reaching this milestone has not been without challenges and hiccups along the way. Some of these challenges were beyond our control: the scheduled power-cuts experienced by all South Africans, also technological challenges that were mechanical and above human control.
Despite these challenges, there were lessons learnt both professionally and personally, and ultimately, we were able to reach our aimed target and move the project to the next stage.
Introduction to the CEE Project
The South African environmental evidence map is a collaborative effort between the South African Centre of the Collaboration for Environmental Evidence (CEE) hosted at the Africa Centre for Evidence, the national government Department of Environment, Forestry and Fisheries (DEFF), and the Gauteng Department of Agriculture and Rural Development (GDARD). The project aims to describe and compile an evidence base of environmental research in South Africa to build a robust reservoir and organised repository of research evidence that may be used by various role players for any questions related to South African environmental policy management and practices. The evidence map is organised around fourteen priority environmental areas, namely biodiversity, waste, air quality, agriculture, land, forestry, fisheries, mining, water, pollution, oceans and coasts, climate, energy, and sustainable development.
Tanya and Metoh joined the project just as searching for relevant studies from academic databases was wrapping up, with searches for grey literature outstanding and screening of search results still to start. The screening of studies to be included in the map is a key step in the production of an evidence map. When the screening process began, we were allocated daily targets of 500 studies per person to screen – that is, read through the title and abstract of each study.
In the first three weeks of the process, we seldom reached 250 studies a day because we were still struggling to make sense of the different areas, research methods used, and other unfamiliar environmental science concepts. However, as time progressed we were not just meeting our daily target but actually exceeding it.
For example, by the eighth week of the screening process we were each able to screen over 600 studies daily on title and abstract. The screening also served the purpose of helping us familarise ourselves with the material on environmental management and practices reviewed in the project. But all rainbows come with some clouds.
Walking the tightrope of search strings and databases
One of the tasks performed was that of grey literature searching. We found that this task of grey literature searches was not as straight-forward as we thought. Searching was very time-consuming, due to the uniqueness of website searches and the need to adapt our searching on each website. A good number of the websites searched did not have published articles in PDF format. We also encountered a few challenges with the academic database searches. For one, we observed that search strings needed to be tailored to suit the different databases searched. And we learnt that academic database searches require exactness and care, as a very slight error can produce faulty outcomes.
Learning the screening software
The software used to house and screen citations for this project is the EPPI Reviewer 4 software, which we were both not very familiar with or accustomed to using. In the beginning of the project, we struggled to find our way around EPPI Reviewer, although as we used it more, it became easier. Even though we encountered a few technical issues, such as software updates which lasted about 30 minutes, much of our initial challenges with EPPI Reviewer disappeared as we became familiar with the software system.
The unpleasant experience of realising how little you know
As mentioned above, this evidence map project is a map consisting of 14 areas across the environmental field, which to an extent is the source of most of the challenges we encountered. We are both trained as social scientists and are now learning evidence synthesis methodologies; we have no formal training in environmental issues, but we have been learning fast!
It was very commonplace for us to read an entire abstract and have no idea what the study was about. This slowed down the screening process as we constantly had to google the meaning of particular concepts to make sense of them in order to proceed with the screening.
However, the more we work on the project, the more we become familiar with the concepts and ‘language’ used in the environmental sector. In the end, the amount of knowledge in the environmental field is remarkable and is one of the key highlights of our experience on this project! Coupled with the professional challenges we encountered on this project were problems of a more personal nature: frequent power-cuts, adapting to working from home during the COVID-19 pandemic, and performance anxiety.
Working in the dark
Living in times where load shedding (i.e., purposeful power cuts by the electricity provider to reduce consumption of electricity) in South Africa is very much a norm in our daily lives. During the time of screening for the South African mega map, this meant that we’d experience constant power-cuts at both our homes and our ACE offices, lasting two hours at a time.
The last quarter of 2020 was particularly difficult: power-cuts lasted two hours, and were sometimes implemented twice daily.
Even though one might have a laptop with battery, during power-cuts we would lose fibre/cable internet access, making it impossible to work on EPPI Reviewer 4. We thus (with ACE support) invested in alternatives, such as buying a dongle to access the internet via 3/4G data, and we also bought power banks to serve as back-up power supply for our devices.
Working from home
Working from home was one of the challenges that did not stem from the project itself, but indirectly and directly impacted the project. We were very nervous about the idea of working from home when we got involved in the project, as working from home can feel isolating. We always wondered if we would ever adapt to the new normal reality of working remotely. However, after more than a year working from home, we have come to enjoy it a lot. In actual fact, we now find working from home very productive as travel time is eliminated. More so, the use of online communication platforms by our organisation, namely Microsoft Teams, ensured that whenever we needed assistance, it was only a call away.
Learning as we do
Performance anxiety was one key challenge that we both experienced at the beginning of our involvement in the project. The anxiety to perform well stemmed from the fact we were not there when the project started and we still have to figure out what it was about.
It also happened at the start of the screening process when we found it difficult to meet the daily target of 500 studies on title and abstract. Eventually, as time progressed, and with support from our supervisor, we began not only meeting the daily target, but surpassing it.
All these challenges have taught us some very valuable lessons from our involvement in this project. These lessons, such as using the classifier or the continuous support from work colleagues, have not only helped in making the process of the project easier for us, but have also managed to make it a learning platform on both professional and personal levels. Below we share these lessons.
Making the most of machine learning: The Classifier
Technologies and methods to speed up the production of evidence synthesis products by reducing the manual labour involved are continuing to emerge. One such technology that was of great use to us in this project was automation. Automation has been proposed or used to expedite most steps of the systematic review process, including search, screening, and data extraction. This came in handy on this particular project given the huge volume of 150 000 studies identified from the searches. In EPPI Reviewer 4, we adopted the bespoke classifier which allowed us to automatically exclude close to 20 000 studies, thereby significantly reducing the manual screening process. As with most of the things in this project, using the classifier was also a first-time experience for us. Our project manager developed a crash course for us to better understand the importance of using the classifier and how it helps. We are also forever grateful from the EPPI support team who were very helpful and responsive to our questions related to the function of the classifier.
Skills enhancement and usefulness of these in our own research
The process of grappling with all the challenges of doing this project has been an exciting one for us. We cannot wait to see the conclusion of the project, and the map! As one of us uses evidence synthesis methodology in our doctoral study, the skills learned in this project have been applied directly in our own research. One of the main takeaways from the process thus far is that it is a continuous process which needs a level of commitment from the beginning, and whilst at times it can be tedious, it is rewarding when you have reached your target.
We have also learnt to appreciate the work that goes on behind the scenes in gathering and synthesising evidence.
Another interesting thing is that because of this process our geographical and environmental knowledge of the world has improved immensely, so 30 Seconds here we come!
The role of support from teammates and supervisors
A final key lesson we learnt is the importance of continuous support from work colleagues. Teammates and supervisors were very supportive throughout the whole process. We cannot overemphasize the support that we received from everyone that we approached at ACE for help. Colleagues were always willing to help explain difficult concepts. A very big thank you to all our colleagues for their team spirit – you live the ACE values!
- scheduled power cuts (also referred to as load shedding) refers to an energy utility’s strategy of reducing demand on the energy grid by temporarily turning off energy delivery to certain geographical areas. Power cuts are only implemented when the electricity generation system has limited capacity to meet all energy demand and is embarked on to optimally manage the available energy in a bid to maintain system stability.
- How to conduct a systematic review, step by step: https://libguides.umn.edu/systematicreviews