TO: Conservation Law Foundation; The Ocean Conservancy; Environmental Defense; World Wildlife Fund Canada; Canadian Parks and Wilderness Society
FROM: Lisa Dropkin, Edge Research
RE:  Response to the Survey Critique
DATE:  March 22, 2002

It has come to my attention that the survey of 750 residents of New England, News Brunswick and Nova Scotia that we recently conducted on your behalf has been publicly criticized in FishNet USA #21.  Criticisms range from the survey methodology and design, to the research objectives behind it, to the integrity of Edge Research in conducting it.  While thoughtful criticism is important for public evaluation of polls and the role public opinion should have in influencing public policy, such criticism should take in the full context of survey design and apply some knowledge of the research industry’s accepted practices.  I thank you for giving me the opportunity to respond.

In Nils Stolpe’s critique of the survey, he states that the purpose of the survey was “supposedly to gauge the level of acceptance of the idea that the public should be willing to accept sacrifices - those associated with an extensive series of no-take zones - in order to ‘save the oceans’.” Actually, the poll was designed to find out what the public believes to be the conditions and problems facing New England and Atlantic Canada’s ocean waters, whether they support the concept of establishing fully protected areas in which no extractive activity is allowed, and how support for such areas is affected if people are told of certain consequences such as a loss of personal recreational access and loss of jobs in the fishing industry.  Rather than asking what the public should be willing to accept, we asked respondents what they think.  And a review of the questionnaire will certainly reveal that an “extensive series” of no-take zones with a rationale of “saving the oceans” was never proposed.

Indeed, while Mr. Stolpe appears to have an issue with conservationists interpreting marine protected areas as “no-take” zones, the survey instrument was explicit in its definition of what respondents were being asked to evaluate.  We did not measure support for “marine protected areas” and then report the findings as support for “no-take” zones.  The question clearly states exactly what would be prohibited and what would be allowed in an area defined as “fully protected” as follows:
In a fully protected area of the ocean, all extractive activities are prohibited, including oil drilling, mining and all commercial and recreational fishing.  Swimming, diving, boating, and research activities are still allowed.  Do you favor or oppose having fully protected areas in (New England/Atlantic Canada’s) ocean waters or don’t you have an opinion on this?

In his critique, Mr. Stolpe frequently ignores the true nature of the questions asked and, for that matter, what most of the poll findings actually said. Here are his criticisms as we understand them and our response:

Stolpe: While the idea of performing research to allow clients to achieve their objectives is certainly understandable from a marketing perspective, it sure isn't science….

Public opinion polling is a tool of social science.  This survey, conducted using random digit dial samples and a sound sampling design meets all industry accepted criteria for statistical validity (such as that of the American Association for Public Opinion Research).

In terms of the purpose of the survey, Mr. Stolpe repeatedly focuses on banning fishing and the use of the poll as a tool to show that New Englanders and Atlantic Canadians want to ban fishing in a large part of the ocean.  This is a gross misinterpretation of the questions asked and the meaning behind the results.  The client articulated a desire to find out what the public-at-large thinks about a resource that is a public trust. In fact, what we found is that public rationale for supporting fully protected areas has little to do with either punishing or helping the fishing industry. It has to do with their sense that it is appropriate for industries of all types to leave certain areas untouched and that there is value in establishing fully protected area for future generations.  Mr. Stolpe raises the point that the survey results do not comport with what fishermen think.  It is important to note that the survey was not designed to interview fishermen about this issue, but rather the public at large.  Fishermen were not excluded from the sample, though they are a small segment (5% of sample in commercial fishing, 6% in any ocean resource dependent job).

Stolpe:  Are the questions in it value-neutral?  In their question examining which factors should take precedence when considering "restricting economic activities in the ocean," respondents could chose between "short term costs in lost jobs, higher prices for goods and services and impacts on families whose livelihood depends on ocean resources" or "long term benefits of healthier and more plentiful resources or fishing and increased tourism to restored ocean places that will improve life for coastal communities and future generations for years to come." Some choice! All things being equal, it's hard to imagine how anyone would choose costs rather than benefits, particularly if the costs were represented as being paid by a specific group (almost undoubtedly a group not represented in the small sample) for a short time, and if the benefits were represented as being accrued by the entire community for both "future generations" and "for years to come….."

Mr. Stolpe’s critique of this question is that it is not value neutral.  Well of course it is not.  It is intended to summarize two opposing points of view to determine how the public weighs these arguments.  This question is asked long after respondents are asked to give their views on the state of the ocean and support for protected areas in value-neutral lines of questioning.  Mr. Stolpe says that the short-term costs are represented as being entirely borne by a specific group (fishermen) however the question clearly states a broader impact of lost jobs and higher prices for goods and services as well as impacts on the families who depend on ocean resources.  About the phrasing of the long-term benefits, Mr. Stolpe says who wouldn’t choose this option?  Well, 23% of Canadians polled did not, nor did 24% of New Englanders.

Stolpe: Is the material in it factual? In the survey the pollsters wrote "Currently, we protect less than 1% of our ocean waters, to preserve this beautiful resource, we need to protect more." The idea that such a miniscule amount of ocean is "protected" would be sure to guarantee that a large proportion of the people polled provided the desired response; that more of the ocean needed to be protected (and so responded 62% of the Canadians and 53% of the New Englanders). But, as anyone who has even a rudimentary knowledge of fisheries management off New England and Atlantic Canada knows, far more than 1% of these waters are already protected. Tens of thousands of square miles are closed to all or to particular types of fishing either permanently or seasonally. These closures, which are in place to protect particular fish stocks, marine mammals, spawning aggregations, migration pathways, sensitive habitat, research areas, etc., etc. affect scallopers, groundfish fishermen, longliners, gillnetters, recreational anglers and pot/trap fishermen. But it's a fairly safe bet that it's easier to sell the idea of protecting more of the ocean from fishing once you've made the case that virtually none of the ocean is presently protected, isn't it? So, regardless of the actual facts, that's the case that was made.

Mr. Stolpe takes issue with the factual accuracy of whether less than 1% of the ocean is fully protected.  This piece of information was supplied to us as commonly accepted accounting of the percentage of the ocean under such protection.  What is far more important to note however, is this question is not supplied to respondents prior to measuring their support for fully protected areas.  Rather, it is one of a series of statements at the end of the survey, about which respondents were asked how persuasive they are.  It is the very nature of opinion to agree or disagree with something regardless of its factual basis.

Stolpe:  Are the results interpreted accurately? The respondents were asked to rate the overall health of the ocean and the commercial fishing industry locally (New England or Atlantic Canada). The possible choices were Excellent, Good, Fair and Poor (or Don't Know). On these questions the pollsters wrote "Regionally, residents are divided in their assessment of the overall health of the ocean: 46% rate it positively (5% excellent, 41% good) and 43% rate it negatively (36% only fair, 7% poor)." Then, regarding the commercial fishing industry, "59% say the health of the fishery is in only 'fair-to-poor' shape compared to 28% who think it is in good shape."

The pollsters at Edge Research - or, as is becoming increasingly evident, "marketers" is a much more appropriate description - have arbitrarily (and kind of amazingly) decided that the "fair" responses belonged in the negative category. This goes against any use of the word "fair" that we're familiar with, but, to be on the safe side, we checked our understanding of the meaning of the word with the definitions offered in several dictionaries. In its context in the survey, "fair" is defined as "adequate" or "average" or, and this might be stretching a bit, "sufficient but not ample." In no way do any of the definitions we came across indicate anything remotely approaching negative or substandard. And it's impossible to imagine that the respondent's understanding of the English language didn't reflect that. Yet, by arbitrarily using the terms "only fair" and "fair-to-poor" the "pollsters" conveniently interpreted all of the "fair" responses as negative…

The purpose of an interval scale is to standardize responses for statistical analysis.  The scale Mr. Stolpe criticizes here is commonly used by pollsters including the Harris Poll and Ipsos Reid.  Rather than being value laden, these scales are used because respondents are able to apply them consistently.  The standard interpretation of excellent-good-only fair-poor is to categorize excellent and good together as positive and only fair and poor together as negative.  Again, this is standard practice and truly it defies common sense to think the average person interprets “only fair” as positive on a four point scale that moves from positive to negative.  For example, in a recent survey conducted by the Harris Poll (October 2001) President Bush received positive (excellent and good) job ratings of 88% and negative (only fair and poor) ratings of 11%.  These ratings comport with those gathered in an ABC News Washington Post survey (September 2001) which used different wording, asking whether respondents approved or disapproved of the job the President was doing.  This ABC Poll showed President Bush with an approval rating of 86% and a disapproval rating of 12%.  It seems clear that U.S. respondents interpret the excellent-poor scale with excellent and good as positive and only fair and poor as negative.

I would add that far from misrepresenting the results, the percentage responses for each category were reported, rather than just the net categories positive/negative so that reviewers could decide for themselves how to interpret that fact that majority of responses fall into the good and only fair categories.

In closing, surveys are a tool used to derive many things:  A snapshot of public awareness or opinion on an issue at a given point in time; an inquiry into public values and beliefs; an instrument to test messages in support of a certain product or policy, the list goes on.

In this case, the survey was intended to take both a measure of current opinions in a neutral fashion and to test how those opinions can be shaped with additional information.  For that reason the questionnaire carefully proceeds from lines of questioning that seek to uncover what the respondents know or feel to be true to lines of questioning that ask them to evaluate different pieces of information and finally to lines of questioning that are more argumentative in nature.  Taken together the intent of market research is to derive a strategy for communications, for product development, for marketing, etc.  Edge Research does not purport to conduct research for the sake of informing the public.  We are not a news organization. Nor do we independently release the results of research conducted on behalf of a client.  It is left to the client to take that step.  What Mr. Stolpe chooses to imply about Edge Research when he says the firm was hired “for anything but their objectivity” is scurrilous.  We would have little to offer our clients if we only told them what they wanted to hear.  What he chooses to imply in his writing about my professional ethics is more than inappropriate, it is flat out wrong.

Throughout his critique, Mr.Stolpe misrepresents the survey by focusing selectively on a few questions rather than the totality of the results.  Whether he likes the answers or not, people in the region say they support establishing fully protected areas.  They say this before ever hearing any language about the supposed benefits of these areas or the negatives.  I would urge that anyone interested review the results in their totality.  I understand that Mr. Stolpe represents the special interests that would be most directly affected by the establishment of fully-protected areas.  But not liking the answers is not sufficient reason to say the results lack validity and that the people at Edge Research lack professional integrity.