Social surveys remain vital to the work of ONS – but maintaining response rates is a serious issue for statistical agencies worldwide. To preserve their value and relevance, surveys must evolve: how they look, how they work and how they are completed. Here, Laura Wilson reveals how ONS is taking on the challenge.
Regular readers of this blog will already be familiar with the transformation work underway at ONS . The Census and Data Collection Transformation Programme is a big part of that. It unites Census, Social and Business Survey data collection change under one roof with the primary aim of achieving ‘administrative data by default’.
The long-term goal is to reduce our reliance in the first instance upon surveys to collect the data we need and move towards utilising government administrative data sources. The key word is ‘reduce’. Surveys aren’t disappearing – ONS will continue to run surveys to be able to meet the needs of users. But those transformed surveys will be very different from the traditional ones.
The programme aligns with the Government Digital Strategy which is to be ‘Digital by Default’. This means that our surveys will be online first, with traditional used for follow-up. The surveys are also being transformed to ensure that data collection is more efficient and the effort required by the person responding is reduced. Online surveying brings new challenges, operationally and for design, but most importantly it brings opportunities.
ONS is not alone in its pursuit of adding online data collection to its portfolio. We run a well-established, interviewer-led, mixed-mode operation of telephone and face-to-face interviewing for the voluntary social surveys. We are very comfortable doing that. But we now live in a digital world and we’ve been embracing that too.
So ONS is heavily investing in its corporate services and infrastructure to deliver this transformation. This includes integrating our systems and sharing them across business areas – employing the principle of ‘build the thing once’.
Adding online to our portfolio doesn’t come without its challenges, but to do it successfully we must challenge ourselves and how we work. This comes in the form of ‘User- Centred Design’ – that is putting the respondent back at the heart of designing the full end to end survey experience. This isn’t something solely recommended by ONS – the Government Digital Service also prescribes putting the user at the centre within service design and has created a great set of principles for us all to follow. This shift is long overdue and is spreading quickly through UK government service design.
After all, if there are no respondents to our surveys then there are no numbers. So it is important that we design for our respondents too and apply an approach which takes their needs into account: from the moment the letter lands on the doorstep, their experience of using the survey and then the between wave engagement. We’ve blended social research and service design principles to transform the social surveys.
We must not underestimate the importance of the ‘experience’ aspect of the respondent-survey interaction. Introducing self-completion means that we can no longer rely on highly trained interviewers to achieve response and to provide a good experience. To address this, we’ve fundamentally changed how we design our surveys: here’s how:
- Designing mobile first, ensuring our surveys work on smartphones in the first instance. Constraining ourselves to less screen space helps us to produce cleaner questions and highlight where we’re trying to do too much in one question
- Optimising questionnaire content; this may mean different versions of questions according to different modes but our testing reassures us that they are understood in the same way
- Looking at how a respondent thinks, and how they conceptualise a topic, in order to construct the question and flow of the questionnaire. E.g. this has resulted in reordering the questionnaire content to help respondents arrive at their answer and in turn increase their confidence in the answer they’ve given. We’ve called this ‘mental model’ research
- Learning from interviewers and seeking their assurance on our prototypes; our transformation starts with them. We speak with them first to learn about the current problems with the wording and flow and look to address those in our redesigns
- Using a conversational tone and using words that respondents would understand. We are also writing for the average reading age of the UK public. We want to write questions that sound like how you’d talk to family or friends about your situation
- Using behavioural insights to design respondent materials. We only include content that is needed by a respondent and our testing has revealed that less is more. For example, shorter materials are better received.
The resulting questionnaires still meet the fundamental output need of a traditional survey, just in a different way. To learn more about this approach you can read my article in the ONS Survey Methodology Bulletin article 78.
Since October 2016 we’ve qualitatively tested with over 600 members of the public in developing the Social Surveys. In July 2017, we ran an online only quantitative test to learn about response using this approach (without incentives) and were pleased to achieve a response rate of 23%.
Therefore, we’re continuing to put the respondent at the heart of everything we do. Our next big test will examine the data quality of the mixed mode transformed questionnaire for online and face-to-face completion. Findings will be available in early 2019.
In short, there is a lot going on – big change is afoot – and for me as a researcher and my team it is all very exciting! This is a huge opportunity for the statistical system and ONS is grabbing it with both hands.
Laura Wilson is Principal Researcher within Research and Design for Social Surveys Transformation
(A version of this post recently featured in the Social Research Association ‘Research Matters’ members magazine.)