Tom Costley, group director of TNS, offers advice on what to look out for in polling data to be confident that the results are significant
The media devotes acres of coverage to consumer surveys – identifying opinions, attitudes and behaviours on every topic.
The following points are worth taking into account when evaluating whether research findings are of value to you or your business:
- Check which organisation has commissioned the research – it is very unlikely a client will have published findings which run contrary to their policy or position on a topic.
- Many surveys are conducted in-house by organisations – if that is the case, the sample of people surveyed is very likely to be of existing customers and the findings will not be representative of the wider population.
- The choice of data-collection method can have a major influence on the survey results – primarily as a result of the different sample of respondents likely to emerge.
For example, the declining proportion of households with a landline and the increase in call-screening technology means response rates for telephone research have fallen and are unlikely to be much above 20%.
Consequently, it is more difficult to achieve a representative sample of respondents from this approach.
An increasing number of surveys, especially in travel and tourism, use an online approach.
As a majority of people who take holidays are likely to use the internet for planning and booking trips, the use of an online survey is appropriate and, compared to phone and face-to-face surveys, is relatively cost-effective.
However, it should be recognised that an online approach tends to attract a higher response rate from younger respondents and may not be as representative of people in older age groups.
For a truly representative sample, such as required for opinion polls and voting intention surveys, a programme of in-home, face-to-face interviews conducted in a series of sampling points in different parts of the country remains the ideal approach for generating accurate data.
If questions can be inserted in an omnibus survey, where costs are shared across a number of clients, then the costs need not be prohibitive.
At a national level, the majority of opinion surveys are based on sample sizes of about 1,000 – this generates data accurate to +/-3-4% at a 95% confidence level, which means 95 times out of 100 the answers would be within this range.
This level of accuracy is widely accepted as standard in market research.
If there are comparisons between different age groups or different geographical regions within the survey findings, any sample size below 100 should be treated with considerable caution.