However, we felt compelled to try to evaluate how recall might affect our results. In the absence of such an evaluation of the accuracy of recall, we would have had to assume that respondents' responses reflected exactly what occurred during ODS/DS. However, we know that recall of events almost a decade in the past is likely to be imperfect. This chapter examines the extent of this imperfection by comparing the follow-up (recall bias) survey we administered to the main survey data.
We were less concerned with this aspect of recall, as pesticide exposure has not been prominent among Gulf War issues. What it does highlight, however, are the two aspects to remembering: ability and effort. We reviewed the extensive literature on questionnaire design, memory, and recall to avoid where possible the methodological pitfalls to which self-reports of exposure may be prone, and we designed the main survey with the findings from this literature in mind. For example, the literature suggests that easily demarcated events--such as a war--are easily recalled, but mundane day-to-day events--such as pesticide use-- may not be. Our goal was to construct a survey that would aid accurate recall by helping respondents reconstruct the context of their experiences. This included questions regarding attributes of respondents' living and working environments, questions on the kinds of pests they faced, and questions designed to help them reconstruct a timeline of their experiences. These questions were intended to encourage their recall of the day-to-day aspects of their life while in the Gulf region.
We cannot know definitively how well survey respondents recall pesticide exposure. However, the literature suggests that we can gauge the extent to which recall might affect our results by examining how changes in reported use at re-survey vary by certain individual characteristics. These include health status and sensitivity to the issues of pesticide use and Gulf War illnesses, as well as other demographic factors such as education.
To reiterate, we are not able to draw firm conclusions about exposure and recall bias solely using information on health status. If ill respondents report more exposure, for example, this could be interpreted several ways: (1) It could be true; (2) it could be because they have been following the debate and talking to others about their experiences; or (3) it could be that healthy respondents are not interested enough in this issue and therefore do not put the same level of effort into remembering. Our objective in collecting the data is simply to document whether responses vary by current health status, rather than to draw definitive conclusions.
We found evidence of changes in responses overall, with the fraction reporting pesticide types increasing about 13 percent in the re-survey. We did not see strong patterns among the various groups in our data; this includes not only demographic groupings, such as education or rank, but also self-reported health status. However, we also found that people who thought about their pesticide exposure before our survey reported more pesticide use, but their
answers were in fact more stable over time. We interpreted the pattern of differences as an indication that people who had not thought about pests and pesticides since the war were less likely to put as much effort into recalling their experiences for our survey. Answers on how pesticides were used (such as number of sprays used or frequency of use) were stable across surveys. In general, most respondents did not report names of pesticides in either survey, nor did they use most types asked about. This remains the most salient finding across the two surveys.
Average Number of Pests | Change in Average Between Surveys | Average Number of Personal-Use Pesticides | Change in Average Between Surveys | Average Number of Field-Use Types | Change in Average Between Surveys | |
Overall | 5.28 | -0.22* | 0.88 | 0.11* | 1.35 | 0.18** |
Air Force | 5.45 | -0.04 | 0.66 | 0.14* | 1.50 | 0.16 |
Marine Corps | 5.23 | -0.14 | 1.15 | 0.02 | 1.38 | 0.29** |
Army | 5.45 | -0.59** | 0.89 | 0.14 | 1.20 | 0.07 |
Navy | ++ | ++ | ++ | ++ | ++ | ++ |
Caucasian | 5.15 | -0.08 | 0.83 | 0.13** | 1.30 | 0.18** |
African-American | 5.86 | -0.76* | 1.07 | 0.03 | 1.52 | 0.14 |
Other | 5.38 | -0.43 | 0.95 | 0.05 | 1.48 | 0.24 |
Male | 5.37 | -0.21 | 0.91 | 0.11** | 1.41 | 0.22** |
Female | 4.44 | -0.33 | 0.50 | 0.06 | 0.83 | -0.17 |
Active | 5.33 | 0.03 | 0.78 | 0.23** | 1.63 | 0.20 |
Reserves | 4.76 | -0.04 | 1.00 | 0.08 | 1.04 | 0.32* |
Retired | 5.09 | -0.14 | 0.70 | 0.05 | 1.32 | 0.09 |
Civilian | 5.51 | -0.43** | 0.98 | 0.10 | 1.33 | 0.18* |
E-1 to E-5 | 5.52 | -0.32* | 0.92 | 0.12** | 1.33 | 0.20** |
E-6 to E-9 | 5.20 | -0.15 | 0.7 | 0.08 | 1.45 | 0.03 |
Officer | 4.11 | 0.16 | 1.00 | 0.11 | 1.16 | 0.53* |
High school or less | 5.08 | -0.21 | 0.85 | 0.03 | 1.39 | 0.21 |
Some college | 5.79 | -0.13 | 0.86 | 0.18** | 1.35 | 0.15 |
College graduate | 4.61 | -0.42 | 0.95 | 0.13* | 1.29 | 0.18 |
NOTE:
The sample used for estimation was the follow-up sample (n = 193).
**p-value 0.05; *p-value
0.1 from paired t-test of original
average to follow-up average.
++Answer suppressed because there are under 10 cases in the cell.
Aware that, given time, respondent' answers could change, we looked for evidence of systematic bias in their answers. We examined multiple dimensions along which we might expect to see such bias, such as service, pay grade, education, and reported health status. Although the total sample size is 193, dividing the sample to look for subgroup differences is statistically difficult, as the power of the tests is reduced due to small numbers. Thus, we did not necessarily expect to find statistically significant results in this part of the analysis. Instead, we placed more weight on the patterns across subgroups and across outcomes, and there we interpreted the results as lacking evidence of systematic bias among subgroups in the survey sample.
The results of this analysis are shown in Table D.1. Although Army members reported many fewer pest types, they did not exhibit the most change in personal-use pesticides (the Air Force did, percentage-wise) nor of field-use types (the Marine Corps showed the largest percentage change). Similarly, African-American veterans report the largest change in the number of pests, Caucasians the largest change in number of personal-use pesticides, and other races the largest changes in field-use types. More educated respondents remembered relatively more personal-use types in the follow-up survey, whereas less educated respondents remembered relatively more field-use applications. The only group whose answers changed in statistically significant ways for all three variables were junior enlisted personnel (pay grades E-1 to E-5), who remembered fewer pests and more pesticides, both personal and field use. This is somewhat, but not entirely, correlated with age, as younger respondents were more likely to be junior enlisted.
One unusual result we found--likely related to recall bias but not related to the recall survey--was that personnel currently on active duty tended to give names of military pesticides whereas civilians tended to give names of nonmilitary pesticides. That is, named pesticides tended to be related to a respondent's current status. We attribute this differential to recall, with current active duty personnel likely having been more recently aware of military products. In contrast, civilians are less likely to have recently been in contact with military products and more likely to have used or otherwise been in contact with nonmilitary products.
As expected, we found that when excellent was the first response presented, as shown in Table D.2, respondents reported better health on average than when poor was the first response presented: 47 percent replied that their health was excellent or very good when those answers were presented first, compared with 36 percent of the other group. The difference is statistically significant (p = 0.07).
Health Status | Version A (n = 86) |
Version B (n = 107) |
Version C (n = 193) |
Poor | 5.8 | 2.8 | 4.2 |
Fair | 10.5 | 23.4 | 17.6 |
Good | 37.2 | 38.3 | 37.8 |
Very good | 26.7 | 25.2 | 25.9 |
Excellent | 19.8 | 10.3 | 14.5 |
NOTES; Response categories were read aloud to the survey respondent. Version A of the question ordered response categories from excellent to poor; Version B was ordered poor to excellent. The sample used in estimation was the follow-up sample (N=193).
Nonetheless, other survey responses appear to be relatively unaffected by health status and by which version of the question was asked. Respondents in both fair/poor and very good health reported seeing more pests in the initial survey than did respondents in good or excellent health, and those reporting fair/poor health also reported fewer pests in the follow-up survey. We cannot explain this odd pattern, having expected to see a smoother change across categories, and so we interpret this to mean that there is no systematic bias by health status. More important, there were no significant differences by health status or question version for number of personal pesticides used and number of field applications witnessed. This is shown in Table D.3, which reports the coefficients from a regression of number of pests (number of types of use) on the health measures and the version of the question asked.
Health Status | Average Number of Pests | Change in Average Between Surveys | Average Number of Personal-Use Types | Change in Average Between Surveys | Average Number of Field-Use Types | Change in Average Between Surveys |
Poor/fair | 1.23** | -0.77* | 0.16 | -0.08 | 0.10 | -0.03 |
Good | 1.00 | -0.34 | 0.31 | 0.01 | -0.02 | 0.04 |
Very good | 1.06* | -0.59 | 0.18 | -0.03 | -0.05 | 0.09 |
Version A | 0.20* | -0.12 | 0.01 | 0.07 | 0.03 | -0.04 |
Constant | 4.27** | 0.29 | 0.67** | 0.10 | 1.34** | 0.17 |
R-squared | 0.03 | 0.02 | 0.01 | 0.01 | 0.00 | 0.00 |
NOTE:
The sample used for estimation was the follow-up sample (n = 193).
**p-value 0.05, *p-value
0.1.
We also explored how these estimates changed when we included whether someone in poor/fair health had reported being enrolled in a Gulf War Registry. For the most part, the estimates remained similar to those reported above. It was interesting that registrants remembered more pesticides (both personal and field-use), and their answers across surveys were more stable regarding the number of types of personal pesticides they used. This is in keeping with the initial survey's questions about how much respondents had thought about pests and pesticides, with those answering "a lot" reporting more pests and more personal pesticide use; additionally, their answers did not change as much across surveys. As shown in Table D.4, those who reported in the initial survey that they had thought very little ("almost none") about pests and pesticides before the interview (most of the sample--see the Introduction) also reported fewer pesticides types in the second interview. The survey asked difficult-to-remember questions about events eight years before the interview. We suspect that respondents who had not thought about pests and pesticides in the intervening years did not put as much effort into remembering their experiences the first time through the survey as did the rest of the sample.
Average Number of Pests | Change in Average Between Surveys | Average Number of Personal-Use Types | Change in Average Between Surveys | Average Number of Field-Use Types | Change in Average Between Surveys | |
Overall | 5.28 | -0.22* | 0.88 | 0.11** | 1.35 | 0.18** |
Before today, how much have you thought bout your Gulf War experiences in general? | ||||||
A lot | 5.67 | 0.42* | 0.97 | 0.00 | 1.30 | -0.07 |
Some or a little | 5.21 | 0.06 | 0.89 | -0.17** | 1.36 | -0.28** |
Almost none | 4.27 | 0.67 | 0.40 | -0.07 | 1.53 | 0.13 |
Before today, how much have you thought about problems you had with pests, rats, or other pests in the Persian Gulf, and the pesticides you used to get rid of these problems? | ||||||
A lot | 6.00 | 0.65 | 1.18 | -0.29* | 1.53 | -0.18 |
Some or a little | 5.78 | 0.08 | 1.12 | -0.06 | 1.49 | -0.16* |
Almost none | 4.76 | 0.25 | 0.64 | -0.11** | 1.21 | -0.20** |
NOTE:
The sample used for estimation was the follow-up sample (n = 193).
**p-value 0.05; *p-value
0.1 from t-test of original average to
follow-up average.
The results for whether someone named the pesticide in either or both surveys were similar. People who named pesticides in the first survey named fewer pesticides in the second survey. We did not expect that. Yet very few named a spray (the most common personal-use pesticide form) in either survey. Of the 24 who provided a name in the follow-up survey, 96 percent gave the same name. It is easy to lose sight of the fact that 83 percent did not change the number of names they provided across surveys, whether they specified a name or not. In other words, few people remembered pesticides by name and this did not change substantially across the two surveys.
We also asked about the number of personal pesticides respondents used by type and the number of field applications they observed. Personal use appears to be stable when taken as a whole--74 percent reported using exactly the same number of sprays in both surveys, 12 percent reported more sprays, and 12 percent reported fewer sprays.[2] Reported frequency of use also remained stable across surveys. As shown in Table D.5, one interpretation of this result is that field use could be underestimated on average in the main survey. The extent of the difference was statistically significant at the 95 percent significance level or better for traps, pellets, and sprays from trucks. Again we note, however, that most answers did not change, largely because most people reported no field use in either survey.
Reported in Original Survey, But Not in Follow-Up | Answer Did Not Change | Did Not Report in Original Survey But Did in Follow-Up | Change in Average Percentage Reporting Use | |
Animal traps | 6.5 | 80.5 | 13.0 | +12** |
Powders | 3.9 | 89.5 | 6.6 | +30 |
Pellets | 0.6 | 95.5 | 3.9 | +25** |
Aerosol | 7.0 | 85.0 | 8.0 | +4 |
Spray from a truck | 1.6 | 93.6 | 4.8 | +11** |
No-Pest strips | 3.2 | 91.4 | 5.4 | +44 |
NOTE:
We examined only forms for which at least 10 people in the follow-up survey
reported observing field use.
**p-value 0.05.
Although the overall frequency of pesticide use may be somewhat higher than the survey results show, there is no evidence that different pesticides were subject to different levels of recall bias. This was qualitatively true even for field use, which showed varying degrees of change according to the type of application reported. There we found somewhat large percentage changes but low overall reporting, and we note that it is easy to lose track of the fact that a large percentage increase in a small number is still a small number. Therefore, we conclude that the mix of pesticides reported in the main survey does not appear to be misestimated.
[1]This question was used to prompt memories of pesticide use by encouraging respondents to recall why they needed pesticides. We did not expect respondents' answers to change across surveys, and so we use this as a gauge of the magnitude of the change in the pesticide measures.
[2]Ninety-four percent of those who report using a liquid give identical answers across surveys about the number of liquids. The other forms do not have at least 10 people reporting use, and so we do not analyze the answers.