What works to increase female wage labour market participation in LMICs: An ACE-led Systematic Review
Here you will find a brief diary of our systematic review experience that will hopefully illustrate the review process and progress, ACE’s approach to systematic reviewing, and provide reflections from individuals within the review team on systematic reviewing.
At the moment we are busy screening the 14,000+ citations that Alison’s scientific search of the academic literature has generated. Natalie, Mary, Charity, and Luvuyo are using EPPI-reviewer for the joyful task of assessing each citation’s title and abstract for relevance to our review. After 2 weeks, there is only 5,500 to go—piece of cake!
(A snapshot of the screening joy)
Two and a half weeks and 15,185 citations later, we are finally done with screening! Naturally, the screening of the last study was duly celebrated. Thanks to EPPI-Reviewer, we have a detailed record of the inclusion and exclusion decisions for each citation and will be able to account for which study made it into the next stage of the review and why. After a quiet afternoon of post-screening recovery—each team member screened at least 500 studies per day over the last two weeks—we will regroup tomorrow and allocate tasks for Grey Literature searches and data cleaning for the included citations.
(When the last citation was screened by the team)
REVIEW TEAM REFLECTION
Luvuyo Zigana on working on his first systematic review: “The interesting thing about systematic reviews is that unlike processed food you know what goes into them. What I have learned so far, particularly in what I have done, is that the most important thing that one can do in order to change the world is to come with an intervention that will produce the outcome that will benefit people. The difficult thing about systematic reviews is that it requires an attentive mind and patience. My take home message after this week is that systematic review is like a methodology of your dissertation, you just never stop working on it.”
Mary Opondo on Grey Literature searching for a systematic review:
This month started off with the searching for grey literature. This part of research resembles the analogy of broken glass, where all the pieces, including the tiniest shards are pieced up together to create beautiful art. And yes, research is art 🙂 I particularly enjoyed this stage of the research, as it is more involving in that I have to search for abstracts in various websites and screen them before uploading the relevant articles to EPPI reviewer. This stage requires a lot of patience (especially for sites that reject Boolean phrases or for “hidden articles”), concentration, strategy and COFFEE!!! 🙂 One last thing: Did you know that empowerment is a simple, yet multidimensional concept?
Natalie Tannous on the joys of finishing the protocol and screening stage: I had some fun moments with my tasks for this week. First I was helping out on the coding of studies at the abstract and title level. I can say that while coding I’ve been reminded of how frustrating vague abstracts are: I’ve seen more than my fair share of ‘entrepreneurship interventions’ and outcomes described as ‘economic lives’. The second major task I had this week was copyediting the protocol. While painfully slow at saving the large MS Word file each time changes were made, I enjoyed the satisfied feeling ones gets after having organised a room or tidied a less-than-perfectly-tidy desk. #successfulweek
Charity Chisoro on finishing the first review milestones and starting a new project: I will use the analogy of a puzzle to reflect on the tasks that I have been doing on this project. A puzzle has many pieces that fit together and every piece is very important as it completes the full picture. The different tasks that I did from the screening, coding, searching of grey literature to uploading articles onto EPPI represent the different pieces of this puzzle. By putting all the pieces together as a team, the puzzle was complete with a final product of a protocol and evidence map. We were able to meet our targets and reach the first milestones through hard work, commitment, support and teamwork. The late Nelson Mandela once said, “After climbing a great hill, one only finds that there are many more hills to climb.” We started this week with an introduction to ESPA project, I am looking forward to learn more as well as contribute to this “puzzle”.
FIRST MILESTONES REACHED!
Six weeks into our systematic review, we have now reached our first two milestones: the production of an evidence map of all research relevant to our review question as well as a draft review protocol. The protocol lays out the methods we intend to follow in the conduct of our systematic review. Usually, there is little room to later deviate from your protocol and the team invested a lot of effort in being as specific as possible in our approach. We are now awaiting peer-review on the protocol.
The evidence map, on the other hand, intends to facilitate a discussion between the review team and the review’s stakeholders. We see it as an engagement tool to ensure that our review’s scope fits users’ needs. In this, we first conceptualised a framework of all relevant interventions and outcomes that pertain to the research area (i.e. women’s labour market participation). You can have a look at the framework here. We then engaged key stakeholders of our review mainly from a policy context on the usefulness of this framework to their decision-making needs. For instance, we consulted on whether our interventions covered relevant programme areas that enjoy policy attention. Following this engagement, we then revised our framework and used it to map the evidence base on what works for women’s labour market participation. The result is the below evidence map, which highlights the size of the available impact evaluation evidence on different interventions that aim to support women’s labour market participation. Our next step is now to once more engage with all our stakeholders on the implications of this evidence map for the full review’s scope. This engagement will help us to identify which programme areas to focus on and what synthesis methods to use in order to ensure that our systematic review remains policy-relevant.
Our review team is currently engaged in discussions with review stakeholders to set the final scope of the research. We have shared a detailed review protocol with DFID, EPPI, and our advisory group. We have also disseminated the interactive evidence map to review stakeholders including the Department of Women and Department of Planning, Monitoring and Evaluation in South Africa. Our meetings and discussions with DFID and EPPI have been particular helpful. Following a joint viewing of the evidence map, we discussed possible review and synthesis options. The interactive evidence map allowed us to target the available evidence-base more effectively and helped to zoom in the most relevant intervention-outcome configurations. Having identified the configurations of most interest to a policy/decision-making audience, we then outlined what type of synthesis approaches we can offer (e.g. a review of reviews, a narrative synthesis, a meta-analysis, and so forth). We are currently signing off the proposed way forward balancing policy interests in the review scope and rigour within the conduct of the review. Our team in Johannesburg is using this quiet research time to tidy up the data collected for the map and to finetune the evidence mapping software that we developed for this project.
A lot has happened with our women in wage labour review. While our protocol still awaits publication due to Purdah rules, the team has made great progress on full-text screening and data extraction. We now have our final list of 18 includes for the review together (more on this soon) and have concluded data extraction for descriptive findings. In addition, a highlight in May was the visit of Dr Ekwaro Obuku from the Africa Centre for Systematic Reviews and Knowledge Translation at Makerere Univerisity. Ekwaro trained our team on using STATA software for meta-analysis and we had a joint session on familiarising ourselves with the Critical Appraisal tool used to assess the trustworthiness of the included studies.
Zafeer Ravat on the completion of the data extraction stage and PDF uploads: The tasks this month have been great. I think the best part is that we are seeing remarkable progress with the project. It is nice to see how tasks such as coding and screening all fit into the data extraction phase. Data extraction was one of the tasks that I had thoroughly enjoyed over the past month. However, I must admit that the attention to detail was at times draining. Especially when the articles were vague about their intervention and population sampling. The uploading of the PDFs was the other task. This was a very tedious and boring task that I am glad is over! I had to constantly remind myself of its importance just so that I could finish. Overall, I had a really enjoyable month and I am glad about the progress that we have been making.
Charity Chisoro on the completion of data extraction: During the month of April, our main task for the Women’s Economic Empowerment (WEE) project was data extraction on EPPI. This involved reading the full-text of the included studies then extracting data and coding it. I found this task particularly challenging and sometimes felt disconnected with the process though I managed to complete it with a lot of support from the team members. In May, we (Mary and I) had a look at the heat map where we had to input the information from EPPI as well as crosscheck the numbers to ensure that they were accurate. In addition, Mary and I started drafting the evidence map descriptive paper based on the information from the interactive evidence map and protocol. We began with the key message, then developed a mind-map and moved on to the background section of the paper. This paper is still a work in progress as we are now busy with the findings section and we will move on to the methods section. June commenced with the tasks of design features, descriptive results and descriptive findings. In addition, I put together the reference list of excludes at full-text citing the reason of exclusion. This month (July), I compiled the summary table of included studies and did a narrative synthesis of the included studies based on the design features. The last couple of months have been an “eye-opener” for me considering that I learnt a lot whilst I was working on the WEE project. The saying that describes my lesson is by Aristotle who pointed out that, “the whole is greater than the sum of its parts.” My involvement in the various parts of the systematic review revealed to me that teamwork is an essential quality, which produces quality outcomes. The publishing of the WEE project protocol this month (July) is one of my proudest moments thus far. It is a gratifying experience to be a co-author of a published article especially bearing in mind that this is one of my very first publications! I firmly believe that this is just the beginning of greater things ahead and not even the sky can limit us. I will end off with a quote by Neil deGrasse Tyson who said that, “there are no limits when you are surrounded by people who believe in you, or by people whose expectations are not set by the short-sighted attitudes of society, or by people who help to open doors of opportunity, not close them.”
Mary Opondo on completion of data extraction: Have you ever focused all your attention on a particular task only for you to look up and go, ‘Huh? Where did the time go?’ Well, that has been my favorite term of late as the data extraction process has been so involving. Of course, this is not to say that I do not take breaks. I do, in fact quite a lot, ironically. After the screening process and production of the evidence map, the team moved on to the data extraction process for the nineteen studies identified for the systematic review process. This process revolved around extracting the intervention design features of the included studies. Unlike the screening stage where articles were excluded at abstract level, the data extraction process required us to go beyond the surface and literally go through the articles word –for word- to get the design interventions, methods, outcomes, cost benefit amongst others. This was a painstakingly slow and tedious process requiring a keen eye to detail lest you miss out crucial information. The task was daunting at first, as I did not have a clue whether I was doing the right thing. It felt like I was walking through a maze with a blindfold, although I later became better at the task after working out a strategy to help me in the process. I learnt that at times having a bigger picture in the mind does not equal to quickly getting the relevant information. Rather, the fastest way to extract data is by coding the relevant features as you read across the pages.
The next phase of the process involved discussions on the key design features, so as to identify which studies would be used to conduct a Qualitative Comparative Analysis (QCA). I particularly enjoyed this phase as it was interactive. It was a welcomed variation to the serious concentration, with eyes wide open extracting information which at times confused me more than ever. The discussions led to nine studies being selected for the QCA. For the remaining studies we had to write narrative synthesis, with the help of our mentors. I helped in writing two narratives on policies, a process that I really enjoyed as I learnt a lot. I was particularly amazed at how small amendments to policies could result to a ripple positive effect on women’s economic participation.
I have so far learnt that the final product of research or any writing isn’t just a result of sheer luck and instant prowess but a slow, procedural process guided with a lot of consultation, second guessing, confusion, drafts, and of course, the silver lining at the end. Research requires your mind to be present, engaged and focused on the current piece of work you are doing. Do not worry if you do not understand the story, as that is part of the process. Above all, I embrace the experience, and take it one step at a time.
ANOTHER MILESTONE REACHED!
We are very excited that our systematic review protocol is published! You can have a read here.