Table 3 EFFECTS process evaluation data collection tools. CHWs community health workers, M&E monitoring and evaluation, PM project manager
Acronym and form # | Form name | Collected by | Respondent type | Frequency | Framework addressed | Key variables of interest |
|---|---|---|---|---|---|---|
Context | ||||||
VPS | Village programs and services | Outcome evaluators | Village leaders, market chairperson | Baseline and endline | Context | Programs and services previously and currently available at the village level by treatment arm; distance to services; activity level of services; satisfaction with services |
MSQ | Market survey questionnaire | Outcome evaluators | Market vendors | Baseline and every 6 months | Context | Availability and price of key foods at the village level; market diversity; travel time to market from an individual household |
FES | Food expenditures form | EFFECTS participants | Context | |||
PGF | Participation in group activities | Outcome evaluators | EFFECTS participants | Baseline and every 6 months | Context | Presence of groups (livelihood, savings, women’s, youth, religious, etc.) in community; active membership by the participant; participation in food or other aid programs; receipt of nutrition, health, or agriculture information, and source of information |
CCY (1) | Community calendar yearly | Field supervisors | Village and community leaders | Monthly | Context | By month and season, food aid, agricultural, health, nutrition information, key local events, religious/cultural festivals, contacts with other CHWs (captures type and frequency by month) |
Community demand and partnerships | ||||||
CDF (2) | Community demand | PM, M&E officers, field supervisors | Village, district, regional leaders | During village, district, regional, and national meetings and visits | Community demand | Qualitative feedback from village, district, regional leaders, and national leaders on perceptions of the project vis-à-vis national priorities and policies, benefit to Mara communities, filling gaps at the community level, potential for scale-up, any perceived spillover at the community level, including changes at the community level and any community actions taken. Note: community demand is also captured in the CSR (community actions) and in the SNF as part of the outcome evaluation (“would you recommend this intervention to a friend or family member”) |
Quality improvement modifications | ||||||
PMQI (3) | Project manager (PM) quality improvement form | PM | N/A | Quarterly; ad hoc | Quality improvement | Processes made to improve the quality of delivery and content enactment. This includes small adjustments made to enhance or improve the content (e.g., key messages, activities, behavior change strategies), delivery of the intervention (e.g., additional home visits, more or less mixed group sessions) and supportive supervision/mentorship (e.g., frequency, content); rationale for the modification; source of the data to inform the decision |
Fidelity of program—dose delivered | ||||||
CSR (4) | CHW session report | CHWs | CHWs | Completed and uploaded immediately after each session | Dose delivered; also covers aspects of spillover and dose received | Date and duration of session, topics and activities covered (session/sub-session topic), delivery quality (ease of delivery, challenges, organization of materials, sufficient time, able to get participants to speak openly), participant responsiveness (participant understanding of topics, what they learned the most, new surprising information for the participant), and reflection on how to improve the sessions moving forward); spillover—participation of other family members, relatives, or friends in the peer group session; number who left early/did not attend entire session |
SSC (5) | Supportive supervision checklist | Field supervisors | CHWs | 10–15% of CHW sessions observed monthly. Form completed and uploaded immediately after each observed session | Dose delivered; supervision; barriers | Quality of delivery—processes including time management, communication and listening, and coaching skills (specifically for parenting/play and communication), facilitation of group discussion and problem-solving, provision of accurate information and effective use of materials including visual aids, motivation and energy level, respect of participants, assignment and review of homework (from field supervisor perspective) |
Fidelity—dose received | ||||||
PGA (6) | Peer group attendance register | CHWs | Peer group participants | Beginning of each bi-weekly peer group session | Dose received (structure and quantity) | Participant attendance (male and female); can be used to assess equality of reach but depends on demographic data we collect (Purdue/Harvard will have to do this) |
HVLSR (7) | Home visit log and session report | CHWs | Peer group participants | After every home visit is conducted | Dose delivered, dose received (structure/exposure) | For participants who miss group sessions, documents home conducted by CHW, content/messages received during home visits, reasons for missing session, challenges faced by the participant |
TERM (8) | Study termination form for participants | CHWs | Peer group participants | If/when a participant drops out of the study | Dose received (quantity and exposure) | ID # of dropped; CHW, date dropped; reason for withdrawal and duration in the study |
PGE (9) | Participant group evaluation form | M&E officers | Peer group participants | 5% of participants every month | Dose received (exposure and satisfaction); also covers aspects of spillover and community demand | Participant feedback on facilitation and satisfaction (delivery, encouraged to participate openly, topics relevant/important, perception of CHW knowledge, enough time during session, practices that are easy/difficult, likelihood of practicing the behaviors); includes 2 items from SNF that ask: “Have you shared what you discussed in the EFFECTS parents’ sessions with anyone else?” and if yes, “What is this person or persons’ relationship to you?” (SNF will also be administered as part of outcome evaluation with addition question of: “would you recommend this intervention to family/friends?” |
PGSF (refer to outcome forms) | Peer group social support form | Outcome evaluators | Intervention participants | Every 6 months (outcome evaluation) | Dose received (exposure and satisfaction) | A 9-item scale measuring social cohesiveness of peer groups |
Fidelity—training | ||||||
CHW-EF (10) | CHW enrolment and ID form | Field supervisors | CHWs | At enrolment, prior to initial training | Training (quantity/exposure) | Lists CHW names, sex, age, education, years working as CHW, residence and contact info, in enrolment/consent dates, prior trainings per content area and self-rated competency, if CHW has previously facilitated group discussion to lay audience (#), topics CHWs want covered during the training and skills they want to build, CHW ID, group ID supporting, village name, supervisor name |
CHW-TAR (11) | CHW Ttraining attendance register | Training lead facilitator | CHWs | At each day of training | Training (structure/quantity) | Staff ID (lead facilitator, co-facilitator (s)), venue location, dates and number of days, trial arm ID, trial arm name, district, CHW ID, CHW name |
CHW PPT (12) | CHW pre- and post-training tests, by topic | Field supervisors | CHWs | Before and after each training | Training (quality) | CHW knowledge, attitudes, and skills on topics they will facilitate; feedback on satisfaction with the training |
TEF (13) | Training evaluation form | CHWs, field supervisors, M&E officers | CHWs, field supervisors, M&E officers | After each training (main training, refresher trainings) | Training (quality) | Satisfaction with training (e.g., enough space, organization, quality of facilitation, quality of materials, time allocation including for interaction and sharing, number of topics covered) as well as what was learned the most, topics that were challenging, and recommendations for improving the training |
TERM (8) | Study termination form | Field supervisors | CHWs | If/when CHW terminates from study | Training (quantity and exposure) | ID # of dropped; CHW, date dropped; reason for termination. Note: will use the same form as the participant TERM form. |
Fidelity—supervision | ||||||
CMR (14) | CHW meeting report | Field supervisors | CHWs | During/after each review/mentorship meeting | Supervision (structure, quality, and quantity), barriers and facilitators; community demand (i.e., community-level actions taken) | Date of the meeting, duration of the meeting, content covered during the quarterly meeting, e.g., summary of progress, breakthrough opportunities; barriers and facilitators to implementation from CHW and FS perspectives—what is working well, challenges and recommendation for overcoming barriers, areas needing improvement |
PMM (15) | Program manager monthly report | Field supervisors | PM and field supervisors | During/after each monthly review/mentorship meeting | Supervision (structure/quantity, quality) | Date completed/month reported on, staff changes, communication with government entities, any partnerships that developed/occurred) sessions covered that month, CHW completion of sessions, major successes, challenges encountered by FS team, lessons learned, QI changes made (if yes, attach form), calendar events that month (if yes, attach form), other issues to report |