Andrew Darnton: Impossible Evaluation?

NHS Change Day_0NHS Change Day is an annual event that takes place across the entire organisation. Staff go beyond their usual activities, to work closer together, share knowledge, disrupt old routines and try something innovative or unusual. The value and impact of NHS Change Day, and similar actions and events, can be far harder to measure than more conventional policies or initiatives. 

Andrew Darnton talks about his work analysing Change Day, producing great evidence that not only justifies the idea, but also illustrates how unexpectedly valuable it is to the NHS. Change Day had previously been considered impossible to evaluate; using the Revaluation approach Andrew defied that assessment.

 

How we measure social value, and how we can get better at what we do (creating social value – especially if we can’t meaningfully measure it) are two joint preoccupations of the Future of Doing Good report.

At the #futuregood event yesterday, the report’s author Sonia Sodha memorably summed up the state of play by saying “Evaluation is focused on proving not improving”.  This kind of orthodox evaluation is out of step with the new wave of innovation in social change which the times require, and which the Big Lottery Fund’s report is responding to.

The report is framed as a set of questions, but the inadequacies of orthodox evaluation have long been apparent to those of us working in social change.  Last year, as one of Two Andrews (Darnton and Harrison) I took a commission from NHS iQ (‘Improving Quality’) to ‘reimagine evaluation’ in the context of NHS Change Day.  Change Day is the one day in a year when the myriad actions of frontline staff taken in support of patients and their carers are made visible, and celebrated across the NHS.  It also self-identifies as a social movement.

As a distributed network of actors pursuing self-directed change, Change Day defies orthodox evaluation.  In RoI terms, the investment (mostly made up of volunteers’ time and capabilities) is as hard to quantify as the return (eg. in staff motivation, freed up beds, or better outcomes for patients).  In the context of a complex system under considerable stress – here the NHS, but it could just as well be a natural ecosystem, or an urban community – Change Day is precisely the kind of social change ‘programme’ we need.  But until we can measure its full value, in ways which are widely recognised as robust, we will struggle to get this kind of activity mainstreamed, or resourced.

To reveal the full value of Change Day, we worked with the frontline staff in the system using a participative approach based on the premise that they as actors know where the value is (after all, it is their own time they had invested).  We then ‘iterated’ with them on questions of the value of what they had done.  We also encouraged them to ‘cascade’ these questions of value out to those they had engaged: patients and families, peers, managers.  When they had done so, we gathered the revealed value together, and used third party ‘proxy’ data as multipliers (eg. the cost of beds on particular wards) to monetise the value – but importantly, only where actors saw the value to themselves in doing so (eg. to get the attention of the hierarchy).  If all the actors in the system did likewise (with or without our help) the £ value of Change Day would be implausibly large – quite a contrast from a system which until now had been deemed unevaluable, and as such was undervalued.  Unsurprisingly, the chief executive of the NHS is now showing a keen interest.

At the core of the Revaluation approach is its understanding of value.  We insist on measuring in three dimensions to reveal the ‘full value’ of a social movement or action, using the ‘3Cs’: Calculate Calibrate Capacitate:

  • Calculate

Involves manipulating numbers (summing, or converting using proxy data and ‘multipliers’) to arrive at a single figure, usually in £. The dominant metric in orthodox evaluation.

  • Calibrate

Involves judgements about the relative merits (or cost/benefits) of different actions and outcomes. Based on how activists decide where to direct their efforts (and how much effort to make), both as individual decision making and socialised in groups.

  • Capacitate

Involves measuring the capacity of a movement or network, plus the potential of that network to increase its capacity in future, and thus the value it can generate (ie. its emergent qualities). Applies both to networks of people and ideas.

Revaluation also highlights how, in complex systems, seeing the whole is a challenge in itself. This is one reason why we work with actors in the system as co-researchers: value is visible locally, but cannot be seen totally from the ‘centre’. We might also say the same in terms of the relationship between the funded (they know what they are doing and its effects) and the funder (who cannot be so sure, especially at whole-programme level).  Accordingly within each of the three dimensions, we distinguish between visible and invisible value: ‘visible’ being known value (evidence of which is lying around, on the surface) and ‘invisible’ being knowable value (which could be evidenced, given time and resources to iterate, cascade, and aggregate).  And as we all know, changes – and the value they generate – take time to materialise; indeed, often long after the funding period (and the orthodox evaluation) is over.  Finally, the method of measurement corresponds to the reality of how social movements make change: the most effective know how to conceal and when to reveal, and they draw strength from the play between the two states.  If our measurement is to help them grow stronger, we need to work with this dynamic.

The resulting framework for summarising the full value of a movement or action can be expressed as the 6 Box ‘dashboard’ for Revaluation:

6 Box dashboard
6 Box dashboard

 

Academic colleagues have called Revaluation “a paradigm shift in evaluation”; others have simply said it’s “not evaluation”.  Through its different ways of measuring and valuing, it supports paradigm shifts in the systems under enquiry.  It was developed in the context of transforming quality improvement in health and social care.  We are now applying it to new ways of valuing ‘landscape-scale’ approaches to tackling biodiversity loss, in the context of the Welsh Government’s Nature Fund.  And we are currently exploring the potential of Revaluation for repositioning the public arts as a vehicle for collective sense making and social transformation.

A loose collective of practitioners is coming together, and we’d be very interested in growing this network, and testing the method in new areas where social change is urgently needed.

Andrew Darnton

[Andrew Darnton is an independent researcher, specialising in behavioural theory and social change. He works at AD Research & Analysis. He is also an Associate of the Centre for the Evaluation of Complexity Across the Nexus (CECAN) at the University of Surrey. The Change Day 2015 Revaluation reports can be found here; a journal paper on the method is forthcoming (Darnton & Harrison 2016).]

 

What other systems and organisations could benefit from this kind of evaluation method? What types of value are impossible to measure? How does the act of evaluation change the thing it is measuring?  Let us know your thoughts and write us a blog, tweet at @IDontWant1 and #FutureGood or comment below!

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s