Digital public health interventions, like apps to help with weight management or stopping smoking, have the potential to change the way complex public health problems are tackled. Done well, they can reach large numbers of people with effective, evidence-based approaches, at relatively low cost.
Health Secretary Matt Hancock’s vision document – Prevention is better than cure – puts both public health and technology at the centre of the health agenda. It outlines how the health system must harness user-centred technology to improve our services – for both the public and healthcare professionals.
But actually measuring the success of digital health interventions is a new challenge. How can PHE and other health bodies develop a consistent and responsive way to find out which digital health interventions work – and which don’t?
To answer that question, PHE’s digital and research teams are developing a practical approach for evaluating the effectiveness and value of digital public health interventions. This approach is designed to be rigorous, but also flexible enough to respond to the rapid, agile approach used by digital teams.
A person-centred approach to evaluation
A good service should be designed for the needs of its users – whether they’re members of the public, public health professionals or NHS staff.
So, before even thinking about how to evaluate digital public health interventions, we decided to learn more about our potential users and understand their experiences.
We spoke with a group of people from across the academic, digital and public health spaces, who develop or commission digital public health services. They shared their experiences of evaluating evidence, and the challenges and successes they met along the way.
Often, public health professionals working in digital health found themselves “running […] up against a clinical way of evaluating interventions” and having to “find a balance between what’s practically doable and what you as an academic maybe would do”.
Our research indicated that those commissioning, designing and developing digital health interventions have unmet needs for evaluation support – made more urgent by the increasing use of digital technology to deliver public health services. In particular, they felt uncertainty about how to go beyond simple engagement metrics, such as clicks, views, downloads, to measure real change in behaviours and health outcomes.
Supporting teams to develop their evaluation approach
A number of concepts emerged from our conversations. Among those we chose to focus on was making sure evaluation is a central part of the design and delivery of any digital health intervention at PHE.
We quickly turned our initial concepts into paper prototypes, then tested these with teams inside and outside PHE delivering digital health interventions – including NHS Health Check, Every Mind Matters and Healthy Start.
We also refined our approach with contributions from academics and public health professionals across the health system, including NHS England, the National Institute of Health and Care Excellence (NICE), and University College London.
As a result of this work, we’ve begun to build a service that can support public health professionals to develop their own evaluation approaches. This service will give teams a practical way to embed aspects of evaluation – like identifying health outcomes and choosing the relevant health data – into the delivery of a digital health intervention.
Evaluating digital health is a system priority
The value of this work extends beyond PHE to the wider health system. Throughout this project, PHE has contributed to other work across government on the evaluation of digital health interventions.
Working alongside NHS England, NICE, MedCity and DigitalHealth.London, PHE is finalising a new framework to support commissioners and developers, so they can see what good evidence for digital health looks like.
The NICE Evidence Standards Framework for Digital Health Technologies outlines the evidence the NHS needs to see, in terms of clinical effectiveness and economic impact, for digital health technologies to be adopted. These standards will be launched publically soon, and our approach to evaluation will align with these new cross-government standards.
Evaluation matters
PHE aims to become a digital-first organisation. That means using both design and digital technology to improve the public’s health – anticipating the public’s needs and providing personalised tools that help people live healthier, longer lives.
PHE, working together with academia and industry, is leading the way in developing best practice and practical methods for evaluating digital public health interventions
As a result, we are building the capability to incorporate meaningful evaluation into the development of digital public health interventions. By supporting teams to effectively measure the success of their intervention, we will promote better digital services – and ultimately, better health outcomes.
To learn more about this work, or to get involved, get in touch with the project lead, Kassandra Karpathakis (kassandra.karpathakis@phe.gov.uk) or the project sponsor, Felix Greaves (felix.greaves@phe.gov.uk).