Valid and reliable evaluation tools are essential to justify federally funded nutrition-education programs, such as the Expanded Food and Nutrition Education Program (EFNEP) and the Food Stamp Nutrition Education Program (FSNEP). UC Cooperative Extension specialists need dependable and consistent evaluation data to develop curriculum and staff training materials. Evaluation tools can also demonstrate the effectiveness of such programs, and can help nutrition educators tailor them to the specific needs of diverse populations. For example, if evaluation data indicates that participants are consistently practicing certain desirable food behaviors, the educator may adapt the intervention and devote more teaching time to other behaviors that need improvement. In addition, good evaluation data at the county and state levels can help program directors determine how to allocate scarce resources and leverage additional funds to enhance or expand successful programs. Data documenting the successful outcomes of the EFNEP and FSNEP programs has been useful in generating new funding for nutrition and wellness education, as well as research activities.

EFNEP and FSNEP impact

The U.S. Department of Agriculture (USDA) funds EFNEP and FSNEP to provide community nutrition education to families with limited incomes (up to 180% of the poverty level). EFNEP has served California families with minor children since 1969 and currently conducts adult education programs targeting head of households in 16 counties. The target group is preschool children, but families with older children are not denied services. About 10,000 families graduate (with certificates) and are evaluated annually after receiving a minimum of 6 to 8 or more hours of instruction in nutrition, money management and food safety. Also, 10 of the 16 counties have youth programs that provide nutrition education training to teachers in elementary schools with large percentages of children qualifying for free and reduced-priced meals. The FSNEP program, instituted in California in 1994, targets food stamp recipients and applicants in 42 counties, and evaluates the nutrition education received by about 11,000 families annually. Both EFNEP and FSNEP hire and train paraprofessional staff as nutrition education assistants (now called nutrition educators). EFNEP and FSNEP staff collaborate with agencies and groups to teach low-income adults how to plan, purchase and prepare low-cost foods for a nutritious diet.

EFNEP and FSNEP are required by USDA to conduct pre- and post-education evaluations of all adult graduates to determine impacts on dietary practices. Both programs use the most recent Dietary Guidelines for Americans (USDA 2005) as a standard to measure improvement among their graduates. Since 1969, EFNEP has measured change in dietary intake with nutrition indicators derived from one pre- and one post-24-hour dietary recall. The participant is asked to recall and record all the food and drink they consumed during the 24-hour period just prior to the recall. If the recall is conducted one-on-one, the educator asks the person to recall the food items and the time of day or night they were eaten and then records the responses. In a group setting the educator explains how the recall is done and answers questions, and the participant records his or her own responses.

In a randomized, controlled study of more than 600 EFNEP graduates in California, the 24-hour dietary recall method evaluated the impact of 6 to 8 hours of nutrition education using the one-on-one teaching model. This study found significant improvement in fruit and vegetable (P ≤ 0.001) and milk consumption, food preparation, and shopping skills (Del Tredici et al. 1988). Similarly, results from annual evaluations of FSNEP graduates in California showed positive benefits for nutrition education delivered in small group settings (Joy et al. 2001).

Other states have documented that the positive impact of the EFNEP program can be long lasting (Burney and Haughton 2002; Torisky et al. 1989). A Michigan study found that the positive improvements after the EFNEP intervention were maintained for more than 5 years (Nierman 1986); plus, the families saved more money on food than the federal cost of administering the EFNEP program (Burney and Haughton 2002).

New evaluation tools needed

Over the last decade, the adult EFNEP program in California moved from a one-on-one teaching model to a group delivery method with between 6 and 30 participants in a class. Administering the traditional 24-hour dietary recall in a group setting is cumbersome. Data collected from these self-administered dietary recalls may not be complete, because an accurate recall requires specific probing by an interviewer that is not feasible in a group setting. Moreover, since underreporting is common with the dietary recall method, particularly among overweight women, use of this method may be especially problematic in low-income groups with a high prevalence of overweight and obesity (McClelland et al. 2001). With today's emphasis on group education, EFNEP teaching staff views the 24-hour dietary recall as an impediment and have stressed the need for a new evaluation tool.

An alternative to the 24-hour recall, food frequency questionnaires (FFQs), are also difficult and time-consuming to administer in a group (Kristal et al. 1998). FFQs include a list of 100 or more foods, a rough indication of portion size, and a set of frequency options (how often consumed) to choose from. Although brief FFQs are available (15 to 60 questions) these shortened versions may not contain enough food items or details to be useful among the ethnically diverse populations served in California (McClelland et al. 2001). With longer evaluation tools such as the full FFQ, respondent burden is also a major concern among low-literacy populations. Moreover, neither the dietary recall nor the FFQ assesses behavioral changes in food-related practices such as reading nutrition labels, removing skin from chicken, serving vegetables at dinner, eating fruits and vegetables for snacks, managing money or handling food safely. However, these are key behaviors targeted by EFNEP and FSNEP.

We recognized the need for a simple and less time-consuming evaluation tool. At USDA's request, we began a study in 1997 to validate a food behavior checklist (FBC). We developed the California FBC and tested it for validity and reliability to measure dietary quality related to chronic disease. Eleven of the 41 FBC questions were related to fruit and vegetable consumption. Our study focuses on whether the new FBC may be used as an alternative to the 24-hour food recall to assess fruit and vegetable intake among EFNEP and FSNEP participants.

Evaluation tool criteria

In addition to being valid and reliable, a good evaluation tool should be responsive to change, easy to administer and analyze, and culturally relevant with a low respondent burden (). A tool is considered valid or accurate if it yields an estimate that correlates well with an accepted “gold standard” (criterion validity). The validity of a new tool may also be established by comparing it to a known tool that measures a similar concept or behavior (convergent validity). A tool that is lengthy or difficult to administer could detract from the delivery of nutrition education, as well as limit the accuracy of the evaluation data.

A reliable evaluation tool must yield similar results consistently when it is administered to the same people on at least two separate occasions in the absence of an intervention. Another test of reliability, referred to as internal consistency, is a measure of correlations among a group of related questions (within the same tool) on the same scale.

A good tool should also be sensitive enough to detect changes in client behaviors exhibited after completing the educational program. The limited-literacy and culturally diverse groups in the EFNEP and FSNEP programs are other key considerations, particularly if the evaluation tool is self-administered in a group setting.

Developing the checklist

A detailed description of the research methods and findings related to developing the comprehensive FBC appears in earlier publications (Murphy et al. 2001; Townsend et al. 2003). This paper gives an overview of the steps taken to develop a short fruit and vegetable FBC, then test it with a group of low-income women in FSNEP.

Fruit and vegetable questions

The questions included in the fruit and vegetable FBC were based on the subject matter taught in the EFNEP and FSNEP programs and the desired behavior changes to be evaluated (see box, this page).

Validation

Baseline data from the original 1997 study of low-income women (n = 100) receiving 6 hours of nutrition education was used to validate the fruit and vegetable FBC questions (Townsend et al. 2003).

We used three validation methods. First, we correlated fruit and vegetable questions with a biomarker, total serum carotenoids (criterion validity). Carotenoids found mainly in fruits and vegetables are the chief source of vitamin A in North American and European diets. These fat-soluble compounds can be stored and converted to vitamin A inside the body (Kagan 1953). They are absorbed in the small intestine, enter the blood via the lymph system and reach peak concentration in the blood within 3 to 5 hours of ingestion. The concentration of human serum (blood) carotenoids depends largely upon the content of the carotenoids in the diet (Woh1 and Goodhart 1968).

Second, we compared fruit and vegetable intakes, as determined by the FBC and three 24-hour dietary recalls (convergent validity). Finally, we compared nutrient intakes such as vitamins A, C and folate (part of the vitamin B complex), as determined by the FBC and the three dietary recalls (convergent validity).

Applying reliability, other criteria

Cronbach's alpha test was used to determine if the results were internally consistent or reliable (Nunnally and Bernstein 1994). As another means to measure reliability, the FBC questions were administered to a separate group of 44 low-income women on two different occasions 3 weeks apart, with no nutrition education in between. The fruit and vegetable FBC was also evaluated for clarity, literacy level and appropriateness for diverse audiences (non-Latino white, non-Latino black and English-speaking Latino adults).

Randomized, controlled FBC trial. Performance of the short fruit and vegetable FBC was assessed among FSNEP participants in eight California counties: Alameda, Fresno, Los Angeles, Monterey, San Francisco, San Joaquin, San Mateo and Santa Clara. FSNEP staff recruited participants from schools and other community organizations.

At baseline, trained interviewers collected demographic data, three 24-hour dietary recalls and the FBC from 100 English-speaking, low-income women who were eligible for food stamps and had at least one child less than 19 years old living at home. These women were randomly assigned to the intervention group that received 6 hours of FSNEP classes or to a delayed control group that received the classes after the study was completed. The content of the nutrition education classes included the Food Guide Pyramid, food labeling, unit pricing, food selection, meal planning, food-related money management and shopping. (The USDA Food Guide was significantly revised in mid-2005, several years after our study, to stress variety rather than just promoting the consumption of five half-cup servings of fruits and vegetables per day.)

The intervention and control groups were not statistically different for a number of variables: education, monthly income, household size, race/ ethnicity and participation in food assistance programs. However, the women in the intervention group were older than those in the control group, 33.2 ± 8.2 years versus 25.8 ± 4.5 years (P ≤ 0.0001) (table 1).

Complete follow-up data, including three additional 24-hour dietary recalls and a second fruit and vegetable FBC, was gathered for 73 women in the intervention group and 14 women in the control group. A randomly selected intervention subgroup (n = 59) provided a second blood sample for analysis of total serum carotenoids at completion of the educational experience.

FBC versus dietary recall

Paraprofessional staff reported that the fruit and vegetable FBC was easier to administer and required less time than the 24-hour dietary recall. The short FBC initially contained 11 questions related to fruit and vegetable intake. Of these, seven questions were significantly and positively related to both serum carotenoid levels (the “gold standard”) and dietary recall variables () (Murphy et al. 2001). Questions related to serum carotenoid levels included: Do you eat more than one kind of fruit daily? and Do you eat more than one kind of vegetable a day?

These seven questions were significantly correlated with two or more nutrient or food group intakes (such as vitamins A and C) as estimated by the dietary recall method. The dietary recalls were analyzed using the USDA Nutrient Database for Individual Surveys (USDA/Food Survey Research Group, 1995 release, updated to Release 8). Daily servings of fruits and vegetables assessed by the 24-hour dietary recalls were correlated to total serum carotenoids (r = + 0.35, P ≤ 0.01). However, a fruit and vegetable score based on the sum of responses for the seven FBC questions had a higher correlation with serum carotenoids (r = + 0.44, P ≤ 0.001) (Townsend et al. 2003).

The internal consistency of these seven fruit and vegetable questions was also acceptable based on the Cronbach alpha of 0.80. Reliability coefficients for the seven individual questions ranged from 0.35 (P ≤ 0.05) to 0.65 (P ≤ 0.0001). No ethnic or cultural differences were found among the responses to these questions at baseline (Townsend et al. 2003).

Fruit and vegetable intakes

Once the validity of the fruit and vegetable FBC was established, we used it with a group of low-income women participating in nutrition education programs. Before the education experience, most of these women did not meet the Food Guide Pyramid recommendations. Only 27% consumed at least two servings of fruit and about 37% consumed three or more servings of vegetables a day. The fruit and vegetable consumption patterns of these low-income women (a mean of 4.5 servings per day) were very similar to the general U.S. population (4.4 servings, median) (Thompson et al. 2005). These fruit and vegetable figures may seem high, but note that all forms of potatoes — including french fries — were counted as vegetables. In addition, 95% of the women studied consumed less than the recommended 25 grams of fiber per day (Institute of Medicine 2002). Their fat intakes ranged from 29% to 42% of total calories, compared with the recommended range of 25% to 35%.

From baseline to follow-up, the intervention group reported significant improvements in the three fruit questions on the FBC (Townsend et al. 2003). The control group did not report any significant changes for fruits and vegetables (data not shown). However, no significant changes occurred for either group in the number of daily servings of fruits and vegetables, as estimated by the dietary recall method. Since the biochemical indicator (serum carotenoids) tended to increase over time (intervention: + 5.2 μg/dl; controls: + 4.1 μg/dl), the dietary recall method, as administered in this setting, may not have been as sensitive to change as the FBC.

For both the intervention and control groups, the energy intake determined by the dietary recall method was actually lower at follow-up, compared to baseline (intervention: -217 kcal; controls: -456 kcal). This drop over time may have been due to subject fatigue in providing multiple dietary recalls.

Due to limited funds, the number of women remaining throughout the study in the control group was quite small (n = 14), which probably limited the study's power to detect change in fruit or vegetable intakes assessed by the FBC or dietary recall. Use of the valid FBC in an intervention study needs to be replicated with a larger sample. This endeavor will be relatively easy now that the FBC's validity and reliability have been established, and since it is no longer necessary to collect blood serum samples and multiple 24-hour dietary recalls.

Checklist advantages

Based on the correlations with total serum carotenoids, the fruit and vegetable FBC appears to be valid and reliable for the assessment of fruit and vegetable intake. In addition, we found that the fruit and vegetable FBC has a number of advantages over the group-administered dietary recall for evaluating the impact of nutrition education. Overall, the FBC is more user-friendly and less cumbersome for both the clients and nutrition educators. The entire FBC takes less time to administer (10 minutes vs. about 30 minutes), and is less complex and easier to score than the dietary recall. The FBC was simple enough to be self-administered with minimal difficulties.

The FBC questions were reflective of the subject matter taught in EFNEP and FSNEP, and the results were responsive to the program goals and outcome objectives. Preliminary results indicate that the seven-item fruit and vegetable FBC is culturally and ethnically neutral with no differences detected in response patterns among English-speaking white, Latino and black women.

EFNEP and FSNEP would benefit from additional research to determine if the FBC is also a valid and reliable tool for non-English-speaking audiences. Research is currently under way to determine if pictorial versions of the FBC would be useful in evaluating nutrition education impact among limited-English and low-literacy populations.