AgrAbility Virginia Program Evaluation Brief: 2016 Survey Results

Abstract:

The AgrAbility Virginia Program supports the safety, wellness, and quality of life of farmers and the farmworker community in Virginia. AgrAbility Virginia is a partnership between Virginia Tech, Easter Seals UCP, and Virginia Cooperative Extension and is funded by the USDA AgriAbility grant program. AgrAbility Virginia works closely with Virginia's rehabilitation and agricultural service delivery system to increase organizational capacity and provide the best quality education and services for farmers across Virginia. To get a sense of how AgrAbility Virginia is meeting the needs of their participants, a Utilization-Focused summative- mixed methods evaluation was conducted. This included both a survey and in-depth semi- structured interviews. This brief only shares brief results from the survey component of the evaluation. Results indicate there is diverse farmer participation in this program living in many counties across Virginia. Responses illustrated that quality of life outcomes are evident. Eighty- three percent (n=10) of participants indicated that positive changes took place after their involvement in AgrAbility Virginia, as well as, over sixty- nine percent (n=9) indicated satisfaction with AgrAbility Virginia program.

Introduction

Utilization-Focused Evaluation or (UFE) is a type of evaluation to assist stakeholders in refining performance in a program. Patton (2008) defines utilization evaluation as a decision- making framework for enhancing the utility and use of evaluations (Patton, 2008). Patton contends that users generally approve of and use this method of evaluation because they feel they have more “ownership” of the evaluation process and findings. This active engagement also fosters a sense of involvement that can be carried throughout the entire evaluation process.

UFE can and does use many different types of evaluation methods. These types include formative, summative, process, and impact approaches (Scriven, 1996). We used a summative type evaluation. Summative evaluation is mostly used at the end of projects where the findings can be compiled to make changes and/or improvements within the way the program operates (Scriven, 1996). Often, a summative evaluation is designed to answer whether or not a program needs to continue as planned, make improvements, or cancelled altogether (Scriven, 1996).

The purpose of this evaluation was to assess the impact AgrAbility Virginia has had on farmers across the state of Virginia. The results from this survey will be applied to make improvements to the program by identifying the needs of AgrAbility Virginia clients and assessing how well this program is meeting those needs. As a core goal, it is essential to AgrAbility Virginia that farmers and farm workers can begin and continue to farm with dignity and hope. Our priorities include:

• Improving access to appropriate assistive technology;

• Increasing access to trusted information and education resources for farmers and their families;

• Providing targeted support for family caregivers;

• Providing capacity building opportunities for professional educators to best support stakeholders

Methods

This study followed a convergent parallel mixed methods design as described by Creswell and Plano Clark (2011). In this type of study, both the qualtiative and quantitative data are collected and analysed separate from eachother and then examined to find commonalities by interpreting the results (Creswell & Plano Clark, 2011).The team chose to implement this design as a way of holistically evaluating particpants’ experiences with AgrAbility Virginia. The quantitative segment of this evaluation includes the survey designed by the AgrAbility Virginia team. The qualitatve portion of the evaluation included in-depth, semi-structured interviews to capture a deeper depiction (Rubin & Rubin, 2005) of what and how the clients receive services from AgrAbility Virginia.

The survey portion consisted of ten pages and 38 questions. The first section included the purpose of the survey and served as the introduction. It also included instructions on how to fill out the survey. The survey included demographic, Likert scale, and open text-box questions. At the end of the survey, participants were asked if they would be willing to partake in an interview. This interview took place later in the year. Demographic questions were used to get a sense of who our clients were, where they are living and how they are farming. The likert scale was used to measure how well AgrAbility Virginia clients were satisfied. The survey was designed to take no more than 20 to 30 minutes to fill out.

This evaluation process started late in 2016, with the creation of a logic model that included questions we desired to have answered by our clients. With the logic model complete and the then the survey created, our team submitted and received Institutional Review Board (IRB) appraoval through Virginia Tech.

In October of 2016, the survey segment began with a pre-recruitment letter that was disseminated either by email or by phone depending on if the client had email and internet. This pre-recruitment letter was determined to be essential for the principal evaluator to approach clients. Next, the official invite letter, along with the survey, was sent to the clients, either electronically or in the mail. A total of 51 surveys were disseminated including, 32 post-mailed and 19 emailed. Many of the clients had moved and/or changed information without contacting the program. Due to this, only 29 surveys were received. The first reminder was sent by email or a phone call on November 7. The 2nd and final reminder was sent on November 14, 2016. We initially planned to close the survey on November 15, 2016. However, we have left it open for a couple more weeks in hopes that more participants would be able to participate in the survey.

While not the focus of this brief, it is important to note that an in-depth interviews were another method used in this program evaluation. Participants self-identified that they would be willing to participate in an interview on the survey portion of the evaluation. They were contacted only if they indicated they were willing to be interviewed. The interview segment began with a team meeting to discuss which questions to ask based on survey responses. IRB amendments were sought and approved. A recruitment letter was sent to interested program participants on April 7, 2017. Interviews were then arranged and conducted by the graduate student. Though nine participants indicated that they would agree to be interviewed, only two participants were responsive and participated in an interview. See our full program report for more information about these interviews and how they complemented the program evaluation.

Brief Survey Results and Discussion

Survey Response

As stated above, a number of the clients changed their address and/or their phone number. Because of this, only 29 clients were reached out of the original 51. In total there were 16 respondants, 7 (43.75%) sent paper surveys back to Virginia Tech and 9 (56.25%) completed the survey electronically. The total responce rate was 55.17 percent.

Demographics

Participants that completed this survey ranged in age from 41 to 79 years old. Most were 50 or older. The mean age was 60. Five participants were military veterans. A total of 14 responders reported gender: four female and ten male respondents. Race was self-identified as white (n=14), African American (n=1), and American Indian (n=1). As a whole, AgrAbility Virginia participants who chose to partake in this evaluation have been farming for at least 15 years. Most survey participants are rural farmers who have been farming between 21-50 acres or over 200 acres for more than 15 years. These farmers generally raise and sell beef at livestock actions 26-49 miles from their farm and are utilizing family labor for production. They farm over 200 acres of land and have 21-50 acres in production. Their farms are mostly located in rural areas. AgrAbility Virginia clients are located in Bedford (6.67%, n=1), Buckingham (6.67%, n=1), Carroll (6.67%, n=1), Chesterfield (6.67%, n=1), Greensville (6.67%, n=1), Hanover (6.67%, n=1), Highland (6.67%, n=1), Prince George (6.67%, n=1), Pulaski (6.67%, n=1), Roanoke (6.67%, n=1), Surry (13.33%, n=2), Sussex(6.67%, n=1), and Tazewell (6.67%, n=1), counties.

ArAbility Virginia also wanted to get a sense of where program participants sold their products, what kind of products they were selling, how far away they needed to travel to sell their products, and what labor they use. The three highest responses indicated that 42.86% (n=6) were utilizing livestock actions for their products, while 28.57% (n=4) were utilizing commodity markets and farmers’ markets. AgrAbility Virginia clients indicated they were mostly selling beef (42.86% n=6), forage (42.86% n=6), feed grains (21.43% n=3), vegetables (21.43% n=3), and poultry (21.43% n=3). The number one response to the travel involved was 26-49 miles at a rate of 28.57% (n=4). Additionally, 14.29% (n=2) of participants reported traveling 0-15 miles, 50-100 miles, not selling products and selling their own products. Most of our clients (64.29%, n=9) use immediate family for labor.

Technical Assistance Recommendations and Assistive Technologies

From the survey, we learned that particular AgrAbility Virginia recommendations and/or assistive technologies used by program participants included: circular stairs, rollover bar, kneepads, handles on shovels and rakes, extended steps, and a catwalk. Clients further indicated that AgrAbility was useful and made positive changes in their lives. The challenges they continue to face are mostly physical. Few participants were aware of additional resources available to them. However, Cooperative Extension, Virginia State University’s Small Farm Outreach Program, and Soil and Water Conservations programs were listed other resources available to participants.

Tools and Education

AgrAbility Virginia often makes suggestions on the use of modified tools to their clients. We wanted to get a sense of what suggested tools where being utilized. Responses included such tools and technologies as a roll over bar for tractors, steps on tractors to make entrance easier, long handle weeding tools, kneepads, mobile sitting chair, circular stairs on grain-bins, and small hand-tool adaptions.

Educational material is an important aspect of AgrAbility Virginia’s mission. It was critical that we ask about the educational materials we provide. The only educational material mentioned, when asked what AgrAbility Virginia material was utilized, was the AgrAbility Tool Box DVD. Most participants (66.67%, n=10) agree they are aware of what AgrAbility Virginia's services are available them. The majority (71.14%, n=10) were aware of what education materials AgrAbility Virginia provides. Half (50.00%, n=7) of the participants indicated that they neither agreed nor disagreed that they knew of additional resources, including service providers, they can access regarding their health because of AgrAbility Virginia's Program.

Changes as a Result of Participating in the Program

Participants were asked to rate the level of positive changes to their farming practices after participating in the AgrAbility Virginia program. Most clients agreed (33.33%, n=4) or strongly agreed (50.00%, n=6) that positive changes were made to their farming practices because of participating in the AgrAbility Virginia program. The following chart illustrates these results.

Overall Program Impression

Participants were asked to indicate their level of agreement with the statement that information provided by AgrAbility Virginia’s was useful to their operation. Participants agree (41.67%, n=5) or strongly agree (41.67%, n=5) that information provided by AgrAbility Virginia has been useful to them. However, 8.33% (n=1) somewhat disagree and 8.33% (n=1) strongly disagree. The table below gives more information.

Participants also indicated that they neither agreed nor disagreed (35.71% n=5) when asked if they received support they needed to make decisions regarding modifications to their machinery. At the rate of 38.45% (n=5) clients indicated they agreed and strongly agreed that they were confident in the modified tools or machinery adapted from their participation in AgrAbility Virginia. 46.15% (n=6) reported they agreed that with AgrAbility Virginia's assistance, they were more hopeful in their ability to meet their farming goals. The majority of clients agreed (53.85%, n=7) AgrAbility Virginia's services and advice were relevant and appropriate for their needs.

Participants strongly agreed (38.45%, n=5) that their quality of life had also increased due to services provided by AgrAbility Virginia. Most (53.85%, n=7) of AgrAbility Virginia’s clients strongly agree that they would recommend them to other farmers or farm families for assistance. The bulk of the participants (53.85, n=5) agree that they are confident in their ability to continue farming safely and productively. As a whole, it is critical that we understand the satisfaction our participants are having with the program. This will help us when making adjustments. Overall satisfaction responses illustrated 46.15% (n=15%) agreed that they were satisfied with AgrAbility Virginia’s services.

AgrAbility Virginia conducted a Utilization- Focused Summative Evaluation or (UFE) to evaluate and then improve the important work our program was conducting. Participants across the state voluntarily filled out a survey and to participate in an in-depth semi-structured interview. The survey included Likert scale questions and closed and open-ended questions. Overall, the clients are satisfied with AgrAbility Virginia’s assistance and services. They are receiving the assistance needed to modify tools and continue farming safely and productively. Results also indicate participants need more education materials and a better awareness of existing educational materials. Recommendations include updating client list and information; provide more outreach about AgrAbility Virginia and the services they deliver; provide more educational material or make explicit where to find educational material; and conduct future evaluations that include new participants.

References

Creswell, J.W., & Plano Clark, V.L. (2011). Designing and conducting mixed methods research. Los Angeles, CA: Sage Publications, Inc.

Patton, M. Q. (2008). Utilization-focused evaluation. 4th Edition. Thousand Oaks, CA: Sage Publications.

Scriven, M. (1996). Types of evaluation and types of evaluator. American Journal of Evaluation, 17(2), 151-161.

Rubin, H.J. & Rubin, I.S. (2005). Qualitative Interviewing: the art of hearing data. Thousand Oaks, CA: Sage Publications.