BRIAN STEEL AGAINST NEW ZEALAND HERALD
Brian Steel complained to the Press Council about a report published by the New Zealand Herald. He contended that various statistics had been presented and reported inaccurately and that the headline could not be justified by the substance of the report.His complaint is not upheld.
Background
The article appeared in the New Zealand Herald on March 14, 2010. It featured the results of a Herald-Digipoll on the mayoral campaign for the Auckland ‘Super’ City.
It was headlined Brown has big lead over Banks for super mayor. The opening words repeated “a big lead” and the next paragraph detailed that in a head-to-head match-up Brown was on 48.4 per cent, 11.4 percentage points ahead of Banks on 37 per cent.
The scope of the survey was given – 731 respondents across Auckland.
A note at the end of the article explained when the poll had been conducted and that the margin of error was 3.6 per cent.
The Complaint
Mr Steel took issue with the term margin of error. He contended that this should only be used when a probability sample had been carried out and in his view the poll was not true probability sampling.
He disputed that Brown’s lead was “a big lead”. Even if the poll were accepted as a probability sample, hypothetically Brown’s share could lie between 44.8 per cent and 52.0 per cent and Banks’ share could lie between 33.4 per cent and 40.6 per cent. ie there was a chance that Brown might be only at 44.8 per cent and Banks at 40.6 per cent. A possible difference of only 4.2 per cent could not justify “a big lead”.
The newspaper responded promptly to Mr Steel, but, dissatisfied, especially with the comment that “enough detail was provided for readers to make up their own minds”, he took his complaint to the Press Council.
In submissions to the Council, he repeated his concern that the Digipoll survey did not meet the criteria for a valid probability sample.
He questioned the response rate achieved and suggested that this was a very important variable (and one not covered in the Herald report).
He claimed that the Herald report published only the base percentages rather than indicating that the lead might have been somewhere between 4.2 and nearly 19 percentage points. In his view the latter statement would have been more helpful.
He explained that the background to his complaint was “to get media to develop a more responsible attitude to reporting surveys”.
He submitted a great deal of material in support, including the ICC/ESOMAR International Code on Market and Social Research and papers from the National Council on Public Polls (US).
Mr Steel stressed that his complaint was against the newspaper, not Digipoll (although he reiterated his view that the sampling method was a representative sample, not a probability sample).
The Newspaper’s Response
David Hastings, the deputy editor, explained that he had sought information from Digipoll. Their advice was that the survey used “statistically sound probability sampling”. The response rate for this particular poll had been 41 per cent.
He countered the second part of the complaint by suggesting that the difference, even at the smallest possible gap of 4.2, was still “10 per cent in relative terms” – enough for a “big lead”.
Further, there had been enough detail for “readers to make up their own minds”. The base percentages had been given as well as the margin of error and readers could easily work out the lead might possibly have ranged between 4.2 and nearly 19 percentage points.
Later, in submissions to the Press Council, he stressed that readers do understand margin of error – and the way this term was used is standard practice in reporting poll results.
He pointed out that the CEO of Digipoll rejected “in the strongest possible terms” the contention that their survey was not a probability sample.
He understood that the Mr Steel wanted the newspaper to have printed “the gap was 19 points at one extreme and 4.2 at the other” but anyone with basic arithmetic would have been able to calculate that from the report.
Discussion and Decision
First, the Council accepts that much of the material supplied by Mr Steel in support of his complaint would be of value to the print media, especially the paper prepared by the National Council on Public Polls entitled “20 Questions a Journalist Should Ask About Poll Results.”
However, on the issue of whether the poll in question was a valid probability sample the Council could find no evidence that the newspaper and Digipoll were incorrect and therefore the use of margin of error seems justified here.
In any case, the Council agrees with the deputy editor’s contention that this is usual practice for media reporting of polls and that it gives a clear warning that readers should view base percentage figures with some caution.
This part of the complaint is not upheld.
The second part of his complaint, that the newspaper should not have claimed “a big lead” for Brown is more difficult.
As the complainant claims, the gap might have been nearly 19 points at its largest margin and merely 4.2 points at its narrowest margin.
Yet those are the extreme points of a range. They might be possible but the much more probable lies around the mid point and the mid point was an 11.4 percentage point difference between the two candidates.
Moreover, the margin of error was published and readers could work out these outer edges of the range for themselves (though they had to read through to the end to get that information).
Was the New Zealand Herald justified in its claim in the headline and opening to the report?
The paper cited above (20 Questions a Journalist Should Ask about Polls) states that “when the gap between two candidates is equal to or more than twice the error margin . . . you can say with confidence that the poll says Candidate A is clearly leading Candidate B”.
The gap was about three times the error margin and the newspaper’s claim for “a big lead” of Brown over Banks seems both justified and reasonable.
It might have been useful to place the detail about the given margin of error closer to the percentage results for the two candidates and also to publish the response rate for the survey but these are minor issues.
For the reasons outlined above the two parts of this complaint are not upheld.
Press Council members considering this complaint were Barry Paterson (Chairman), Pip Bruce Ferguson, Ruth Buddicom, Sandy Gill, Penny Harding, Keith Lees, Clive Lind, Lynn Scott and Stephen Stewart.
John Roughan took no part in the consideration of this complaint.