AgrAbility Virginia’s Mixed-Method Program Evaluation Approach and Considerations

Introduction

In 2016, the AgrAbility Virginia Program set out to understand how well we were meeting the needs of our clients. To do this, we conducted a mixed-methods program evaluation. In 2017, our data results were recorded, analyzed and reported. Please see AgrAbility Virginia 2016 Evaluation Report for more specifics on the evaluation process and results. This brief details what we learned from designing and implementing our statewide program evaluation.

Conducting a comprehensive evaluation was a vital component of our programming efforts. The intention was for us to get a better understanding of how we are performing, whether we are reaching our goals, and to inform us of our next steps. Ultimately, it was an avenue to assist us in ensuring that we are making sustainable improvements that will lead to the betterment of our overall program and our participants. The purpose of this evaluation was to assess the impact AgrAbility Virginia has had on farmers across the state of Virginia. The results will be applied to make improvements to the program by identifying the needs of AgrAbility Virginia program participants and assessing how well the program is meeting those needs. As a core goal, it is essential to AgrAbility Virginia that farmers and farm workers can begin and continue to farm with dignity.

Our priorities include:

• Improving access to appropriate assistive technology;
• Increasing access to trusted information and education resources for farmers and their families;
• Providing targeted support for family caregivers;
• Providing capacity building opportunities for professional educators to best support stakeholders

Because of our priorities and goals, we choose a Utilization-Focused Evaluation. Utilization-Focused Evaluation or (UFE) is a type of evaluation that was developed to assist stakeholders in refining the function of a program (Patton, 2008). It was critical that we completed this evaluation in a way that allowed it to be used to identify the areas that were being completed well and the areas that needed improvement. This UFE used a summative type evaluation. Summative evaluation is mostly used at the end of the projects so the findings can be used to make changes and/or improvements with in the way the program operates (Scriven 1996). Often, a summative evaluation is constructed to answer whether or not a program needs to continue as designed, make amendments, or abandoned entirely (Scriven 1996). The team also thought it was important to incorporate both qualitative and quantitative methods by using interviews and a survey. Our approach followed a convergent parallel mixed methods design as described by Creswell and Plano Clark (2011).

Organizing the Evaluation

Completing a successful evaluation began with organizing our team. We initially met to set up future meetings to address evaluation strategies, questions, and revisions to the design. The gatekeeper for reaching our clients was quickly identified as the AgrAbility Virginia Program Coordinator since he had direct contact with the clients. With these connections, it seemed to be a natural fit for the Program Coordinator to introduce the evaluation and its purpose. Our first meeting was held to discuss our mission statement and what questions we wanted to ask our clients. We came up with a logic model and soon after began to narrow down are key questions. With the logic model complete and the survey created, our team members began the process of becoming Institutional Review Board (IRB) certified through Virginia Tech. We also receive approval for distribution of the survey.

Conducting and Amending the Evaluation

The survey segment began with a pre-recruitment letter that was disseminated either by email or by phone (depending on if the client had email and internet) on October 24 -28, 2016. The invite letters along with the surveys were sent to the clients, either electronically or in the mail on October 28, 2016. A total of 51 surveys disseminated, including 32 post-mailed and 19 emailed. The first reminder was sent by email or a phone call on November 7. The 2nd and final reminder was sent on November 14, 2016. However, the survey was left open for two additional weeks to allow more time for participants to fill it out.

The interview segment of this evaluation began with a team meeting to discuss which questions we wanted to ask in a more in-depth way. IRB amendments were sought and appliqued. A recruitment letter was sent on April 7, 2017. Interviews were then arranged and conducted by the graduate research student. Though nine participants indicated that they would agree to be interviewed, only two participants were responsive and participated in an interview.

Program Evaluation Insights and Considerations

As a team, we faced a few hurtles. These included research and client challenges. As a team, we looked for ways to work through each. Not all of these were solved and some were noted as areas we needed to improve or consider altering with future evaluations.

The first big challenge we faced as a team was finding time to meet and come up with a logic model. We all follow completely different types of schedules, to include: professor, student, full and part time employment hours. The AgrAbility Virginia team is spread out across the Commonwealth of Virginia and do not work out of the same central offices.

However, proper schedule management, communication, and consistent phone call meetings are effective strategies we used to keep the process moving forward.

Another challenge was obtaining IRB approval. AgrAbility Virginia is a relationship between Easter Seals UCP and Virginia Tech. It was important to get all members of the team IRB qualified. Although, some members of the team were able to, one member of our team was not able to access the training and therefore did not become IRB certified. Unfortunately, he was not able to access the data and provide input regarding the direct findings.

AgrAbility Virginia clientele is small but critical. Our clients experience some type of physical injury, illness, or disability. Contact with them ought to be culturally sensitive to their needs. Some of our clients do not have internet, which requires reaching out in person or on the phone. Additionally, several of our clients changed their phone numbers and even moved to new addresses. This made communication challenging and in some cases, impossible. As a result, some clients did not receive a survey and fewer clients we able to include their recommendations and comments. We learned that our information needed to be updated more regularly for future evaluations. Overall, 29 out of the original 51 were reached. In total, there were 16 respondants, 7 (43.75%) that sent paper surveys back to Virginia Tech and 9 (56.25%) that completed the survey electronically. Making the total responce rate 55.17 percent.

The AgrAbility Virginia team experienced a number of positive outcomes while completing this evaluation. For one, we discovered that as a team, we incorporated and developed individual skills that were needed to fully complete this task. We realized that we would need the skill set of the entire team to gain an accurate perspective. We also reinforced our already established knowledge of our ability to work together to develop an even better result. This being assessment questions and eventually a completed evaluation. We were able to focus in on what needed to be asked so, to better our program and everyday educational services.

Final Thoughts and Next Steps

As the process of this evaluation moved forward, we were able to see areas we needed to improve on and areas we were conducting well. The process reinforced to us that we have a good team that serves Virginia farmers with success. With the data we acquired, we will be looking toward the future to make improvements or enhance the areas we perform well. In particular, we agreed that it is important to improve communication with our participants and regularly update participants’ information. One of our top priorities will be to improving our outreach and increase the number and type of educational materials produced. Additionally, we will make this evaluation an annual practice.

References

Creswell, J.W., & Plano Clark, V.L. (2011). Designing and conducting mixed methods research. Los Angeles, CA: Sage Publications, Inc.

Patton, M. Q. (2008). Utilization-focused evaluation. 4th Edition. Thousand Oaks, CA: Sage Publications.

Scriven, M. (1996). Types of evaluation and types of evaluator. American Journal of Evaluation, 17(2), 151-161.

Rubin, H.J. & Rubin, I.S. (2005). Qualitative Interviewing: The art of hearing data. Thousand Oaks, CA: Sage Publications
AgrAbility Virginia Program

AgrAbility Virginia Program is funded by the AgrAbility Project, USDA National Institute of Food and Agriculture (NIFA), Special Project 2014-2018 (41590-22326). Administered by Virginia Tech, Easter Seals UCP North Carolina & Virginia, Inc. and Virginia Cooperative Extension. www.agrabilityvirginia.org.