As SPM Pros (SPI qualified auditors, Client Protection assessors and trainers, TA providers) we value being in the field, organizing face-to face workshops and trainings, conducting interviews, meeting with FSPs’ staff and clients, discussing pros and cons, sharing a coffee… This is how we manage to make direct observations, get in-depth insights, and cross-check information. It helps us get a good understanding of the context, the issues at stake, the relationships between various stakeholders, the gaps and needs to be fulfilled. This is also how we manage to provide tailored support, build local capacities, help embed good practices, support improvements, and create buy-in for SPM.
The COVID-19 pandemic has been quite a challenge for all of us, limiting opportunities for direct interactions. We had to adapt, and, after a while, many of us started to conduct evaluations and provide support remotely.
More than a year after the beginning of the pandemic, we thought it would be time to share some insights on remote evaluation experiences. To do so, we asked for your contributions, and received detailed inputs from 9 SPM Pros, who have conducted remote evaluations during the past year*. We summarize here your key insights.
* SPI4 audits, Client Protection Assessment, Client Protection Certification, social rating, due diligence, or project evaluations.
The rise of remote assessments, despite our preference for in-field presence
In the early months of the pandemic, CERISE communicated on a limited number of situations where we felt remote audits could be justified (a strict deadline to be met, a repeat audit, or in cases where the consultant had already worked with the FSP). But as time went on, we saw more and more requests from partners and their governance bodies for external assessments: in some cases it was the deadline for Smart Campaign Certification label, in others, a desire to understand clients’ needs and preferences in the context of the sanitary crisis, or just a desire to continue operations, in spite of it.
What changes in a remote evaluation?
Experiences have been quite diverse. For a majority of respondents, conducting the evaluation remotely extended the overall length of the mission. This is mostly due to the fact that it was more difficult to get access to people and information, compared to in-person missions. There was often more time spent on re-scheduling interviews and sending emails back-and-forth until getting the right information. As Roberto Ibarra Rivas shared: “By working remotely, the agendas set were frequently changed by the people to be interviewed, which really extended the interview process.”
For a few SPM Pros, conducting the evaluation remotely actually had the reverse effect: it shortened the overall duration of the mission, as there was no more time spent travelling, or as part of the mission was delegated to local consultants.
Be the missions shorter or longer, many respondents estimated that conducting evaluations remotely did not reduce the level of quality compared to in-person evaluations. They praised the existence of visio-conferences tools that allowed them to conduct interviews in a relatively lively way, while acknowledging that one first needs to get familiar with such tools to get the most out of them. Carla Aguirre Marinovic even considers that “in a certain way, it improved the service provided, since I was able to give additional technical assistance to the institutions, in a tailored way.” Remote assessments can indeed be conducted on a longer period of time, with regular calls adapted to the time constraints, capacity of integration, and agenda of the institution, instead of concentrated during an intensive week of field visit.
The convenience of using digital tools does not seem to be the same everywhere though. While SPM Pros working with Latin American FSPs seem to have found a good alternative by using digital tools, it was less the case for SPM Pros working with institutions in Africa or Asia. Lack of connectivity is a first reason that prevented them from conducting all interviews as initially planned or from receiving expected documentation. Language could be another issue, as Kanchan Gidwani mentioned: “A lot gets lost in translation in the virtual calls.” For some SPM Pros, the quality of the mission was thus negatively impacted by the fact that it had to be conducted remotely. As Anne-Laure Behaghel shared: “I could only interview the management and the Board, nobody from the field, even less clients. Moreover, we have to rely much more on declaratives, as information cross-checking is far more difficult if we ask additional information by email, which we do not always get.”
The risk of remote assessment is also to miss key information that would be directly available in a face-to-face meeting: for example, not realizing the organization has plenty of information available in its MIS, when they can only share simple, basic excel extractions of data by email.
Conducting interviews
To conduct interviews with FSP staff, interviewed SPM Pros mainly used the phone or visio-conference tools such as Zoom, Teams, Skype or Whatsapp. However, whenever the context allowed it, they tended to go in the field and conduct in-person interviews. That was particularly the case for SPM Pros who were based in the country where the mission was conducted. For Dr Tej Hari Ghimire, “going to the field was also a good opportunity to deliver awareness on protection measures to be taken to be protected from COVID-19 in remote rural areas.” This mixed approach enabled most respondents to interview staff at both head office and branch levels: “it was a good experience, the mixed approach worked well” shared Edgar Juárez Sepúlveda. Although, in other contexts (lack of connectivity, no possibility to go in the field), SPM Pros had to give up on interviewing staff at branch level.
For interviews with clients, the situation is also contrasted. Some SPM Pros were able to conduct interviews through video-calls; some others waited for the pandemic curve to go down to still conduct onsite interviews; a few experts worked with local consultants who conducted the interviews in-person whenever possible; while others could not conduct any interview with clients, for lack of connectivity and lockdown restrictions.
Crystal, a FSP in Georgia that CERISE is working with, had an interesting experience in conducting remote interviews with clients. In the frame of an impact assessment, quantitative surveys with clients were conducted over the phone; but for in-depth qualitative surveys, interviews were conducted with Microsoft Teams. The organization managed to conduct 45 interviews of one hour on average. The low level of internet usage among customers aged over 50, especially in the rural areas, was a major constraint, and younger family members or regional staff of the organization were involved, to guide the clients in installing Teams and getting connected. Once the technical issue was resolved, the quality of the interview was high, with the clients valuing being listened, and the soft barrier of the computer allowing for transparent, genuine and frank discussions.
Picture below: online qualitative interviews with Crystal clients by Natia Katsiashvili (at the center).
Adjusting methodologies
Conducting evaluations remotely quite logically implied adjustments to the methodology used. In some cases, sampling had to be reviewed to focus on FSP branches with connectivity (with a consequent bias in the sample selection). Other SPM Pros mentioned that, to compensate for the lack of direct observation, they had to go more in-depth in the interviews and asked for additional documents.
Some adapted their methodology by working more closely with the FSP, involving staff in the evaluation design, data collection and data analysis. Aïda Gueye mentioned: “I ended up spending more time working with SPM Champions.” For Anne-Laure Behaghel, the approach was “more that of an accompanied self-assessment, rather than an external audit.” In the case of Crystal, instead of visiting the organization for 2 on-site focused, time-bounded missions, the work was spread over several months, with almost systematic weekly calls. This approach allowed a better appropriation by the local team, a more efficient capacity-building and experience-sharing, which gave the organization confidence to conduct the client assessment on its own for the next survey.
Any tips?
SPM Pros have adapted to the COVID-19 context with flexibility and creativity. Conducting evaluations remotely was a necessity to keep the work going on; but whenever it was possible, preference was still to conduct in-person interviews and workshops.
As it is likely that in-person missions will still be limited in the coming months, we asked SPM Pros if they had any tips to share. Here are some of their answers:
- “Learn how to use remote digital tools…” – Javier Vaca
- “Set an agenda with exact timetables […] and respect the timetables!” – Jessica Herrera
- “Work with SPM Champions and local consultants” – Aïda Gueye
What about you? Would you have any tip to share with the community?
Please send them to us so that we can regularly update this blog with additional inputs!
