The earth is round (p<.05) – Personal health budgets and randomised controlled trials

The title of this post is from one of my favourite statistical thinkers (don’t judge me!), Jacob Cohen¹, in an article arguing that the logic of statistical hypothesis testing (the p<0.05 bit) simply doesn’t fit many important research questions, and that making a fetish of statistical hypothesis testing may actually impede sensible research that can make a useful difference to the world.

“Consider the following: A colleague approaches me with a statistical problem. He believes that a generally rare disease does not exist at all in a given population, hence H₀: P=0. He draws a more or less random sample of 30 cases from this population and finds that one of the cases has the disease, hence P_s=1/30=.033. He is not sure how to test H₀, chi-square with Yates’s (1951) correction or the Fisher exact test, and wonders whether he has enough power. Would you believe it? And would you believe that if he tried to publish this result without a significance test, one or more reviewers might complain? It could happen.”

Why am I telling you this? Well, with the announcement of a rapid expansion of personal health budgets have come repeated and influential calls for randomised controlled trials (RCTs) to test whether personal health budgets ‘work’ or not. My concern is that the RCT is becoming increasingly fetishised as the only valid research methodology, and applying an RCT methodology to the expansion of personal health budgets may not be the most effective use of scarce research funding to address the essential research questions. I’m writing not as a person who is opposed to RCT methodologies generally (I’m involved in doing one at the moment), and not as a person who thinks the expansion of personal health budgets shouldn’t be accompanied by rigorous evaluation. I should make clear that I’ve been involved in the collection of practice-based evidence concerning the early development of personal health budgets³, but as the report makes clear I see this as complementary to rigorous research on the issue.

The NHS Choices website, in their news glossary, provides a useful short definition of an RCT (and a judgement too):

“This is a study where people are randomly allocated to received (or not receive) a particular intervention (this could be two different treatments or one treatment and a placebo). This is the best type of study design to determine whether a treatment is effective.”

The logic of RCTs is simple, and has its roots in testing drugs but has much wider scope. You have a treatment X, and you want to see if it’s better, worse or the same in its impact than another treatment Y. Because we often (secretly or openly) want one of the treatments to ‘work’ better than the other one, we can’t trust ourselves to allocate people fairly across the two treatments (without even realising it, we might allocate all the people who we think might be more receptive to the treatment into our favoured condition). So we randomise people instead (in essence on the flip of a computerised coin) so that we can have no influence on who gets what treatment. We hope that this will result in two groups of people that will, on average, turn out to be evenly matched on anything that matters, although randomisation doesn’t guarantee this.

Sounds simple, and indeed the logic of the RCT is straightforward. So why don’t I think it’s the right way to evaluate a rapid expansion of personal health budgets?

First, RCTs assume that there is a more or less standard ‘intervention’, with an agreed standard ‘outcome’ (what the intervention is trying to have a positive impact upon). For me, there are a number of problems with applying this logic to personal health budgets:

·         The way I see them, personal health budgets in themselves are not an ‘intervention’ or ‘treatment’: they’re a different way of commissioning. This means that the ‘treatment’ is in principle not separable from the complex systems within which personal health budgets are embedded. This is important as RCTs typically start with what are called ‘efficacy’ trials (does the intervention work under ideal and controlled circumstances). If these show that the intervention is effective, then RCTs should (although these often don’t happen) to ‘effectiveness’ trials (does the intervention work in the real world), where unsurprisingly the effectiveness of the intervention is much smaller than under ideal conditions. For me, the nature of personal health budgets mean that an ‘efficacy’ trial is either in principle impossible, or at the very least would denature the personal health budgets so severely (in the name of creating standardised, controlled conditions) to make the findings meaningless. So, arguments that ‘proof of concept’ for personal health budgets needs to be established before their application in the real world are misplaced.

·         Related to this, personal health budgets will be a moving target. Because they involve complex changes (and challenges) to existing systems, the ways in which people can use them (and how receptive systems are to them) will be very different in 2015 than in 2020, say. Conducting an RCT that will be out of date by the time it’s completed, as the personal health budget ‘treatment’ will be quite different by then, severely reduces its utility.

·         Personal health budgets will be applied to hugely diverse groups of people with hugely diverse health needs who will use personal health budgets in hugely diverse ways, with hugely diverse parts of health and other systems. Such diversity is a real issue for RCTs – is a personal health budget for a person with mental health issues the same ‘treatment’ as one for a person with COPD, for example? (and variation usually means needing bigger and bigger numbers of participants)

·         Personal health budget holders will be using personal health budgets to achieve highly personalised outcomes for them. Any ‘primary outcome’ measure will have to be a measure of how people are meeting their individualised goals, rather than a standard health or quality of life type outcome measure, especially as for many people the goal of a personal health budget may be to preserve/maintain valued aspects of their lives for as long as possible rather than seeking to improve health/quality of life outcomes (back to diversity again).

·         The logic of the RCT requires that the form of the treatment must be standard for all research participants across the course of the project – this means that any learning about how they could be done better must be resolutely ignored by/withheld from participants while they are involved in the RCT.

·         The RCT requires a fairly standard comparison/control group that is completely separate to the treatment group. In the case of personal health budgets, what would this be, particularly as ‘treatment as usual’ is likely to change as a result of the impact of personal health budgets on health systems, even if a person isn’t getting a personal health budget.

·         For me, the RCT research question ‘Does Treatment X work better than Treatment Y?’ is not the most urgent or important question. When, how, why, in what ways, for whom do personal health budgets work (or not), and what can we learn as we go to improve the scope and coverage of positive impacts of personal health budgets, seem a much more relevant set of questions to me.

Second, having been involved in conducting a relatively small RCT, I’m acutely aware of the time RCTs take, the (potentially necessary?) bureaucracy that surrounds them, and how expensive they are. Even if my concerns above were ill-founded, there are good pragmatic reasons why an RCT would not be the best use of scarce research resources:

·         RCTs take a long time! Setting them up takes a while, decisions would need to be taken on what would be a sensible timescale for personal health budgets to ‘work’, and there would possibly be follow-ups too? As I’ve said above, I think this would mean that RCT findings would be assessing a historical moment in time by the time they came out.

·         The standardised nature of the RCT method would denature the actual application of personal health budgets over time, both for trial participants and for the service systems with which people engage.

·         As personal health budgets expand, I’m unclear how people would react to being randomised, particularly if they’re randomised to the ‘treatment as usual’ condition.

·         Methodologically, the risk of ‘contamination’ (charming word) across conditions is huge - people (even health professionals!) talk to each other which could compromise the RCT trial design in all sorts of ways.

·         With the diversity of people’s health needs, how big would any RCT have to be? Across how many areas of the country? How expensive would this get? This seems to me like we’re entering into the realms of ‘waste’ in research (http://www.thelancet.com/series/research).

So, what are the alternatives to RCT methods in this context? Personally, I think the research funding that is available could be better spent on a programme of implementation research⁴, with people using personal health budgets and those supporting them at the heart of decision-making about what the essential research questions are, and which research methods will yield relevant, timely findings that can feed into ongoing learning about how to do personal health budgets well. Too many good ideas have suffered when they have attempted to be scaled up – why not design a research programme that rigorously investigates such an attempt to scale up, while simultaneously supporting it?

References

1.            Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 12, 997-1003. http://ist-socrates.berkeley.edu/~maccoun/PP279_Cohen1.pdf

2.            NHS Choices. News glossary. http://www.nhs.uk/news/Pages/Newsglossary.aspx

3.            Waters, J. & Hatton, C. (2014). The POET surveys 2014. Personal health budget holders and family carers. London: Think Local, Act Personal. http://www.thinklocalactpersonal.org.uk/_library/Resources/SDS/POET_health_FINAL_24_Oct.pdf

4.            Peters, D.H., Tran, N.T. & Adam, T. (2013). Implementation research in health: A practical guide. Geneva: Alliance for Health Policy and Systems Research, World Health Organization. http://who.int/alliance-hpsr/alliancehpsr_irpguide.pdf

My kind of high security

In my ‘Return of the Chunky Ninjas’ blogpost (http://chrishatton.blogspot.co.uk/2014/10/return-of-chunky-ninjas.html) a couple of weeks ago, I suggested the following:

Rather than having block contracts for big service providers, why not have guaranteed 10-year amounts for people’s IPCs (people could return some of their IPC if they thought it was too much, and there would be the option for the amount of the IPC to increase if necessary). The money always follows the person.

It got a bit lost in subsequent Twitter discussions, except for @neilmcrowther who commented  “imagine could make huge difference if someone got that age 14-24”

To be honest, I was a bit embarrassed that this thought had occurred to me some 20 years later than it should have done. But again, as @neilmcrowther commented “cos something that should be so intuitive runs completely counter to common custom?”

So, this is a quick blogpost to highlight this idea and hopefully get some discussion on it. Why do I think it could be helpful?

1) It could remove at one stroke a lot of the insecurity that people and family members have to experience about the amount of their personal budget, particularly in terms of arbitrary changes/cuts to the amount and the feeling of also having to justify yourself to the local authority/NHS all the time.

2) This security gives people the chance to really plan for the ‘imagined future’ (@sarasiobhan’s vital words), rather than being stuck in a perpetual, insecure present.

3) It places more power in the hands (both individually and collectively) of people, families and allies to shape what supports they want in their local area.

4) It also removes a lot of the petty and spiteful bureaucracy (I’m thinking particularly of the experiences of @MarkNeary1) that often seems to unnecessarily attend the way that local authorities administer personal budgets. Why not leave people to get on with it, with people simply producing a straightforward annual budget report every year?

5) It separates what should be a positive process of helping people work out what supports will work best for them from what is felt as a punitive process of deciding how much the budget should be, which I imagine would be a great relief to many social workers as well as to people and families.

Of course there are issues with this idea. The initial budget setting process obviously assumes huge importance. There also needs to be some way of checking for outright fraud that isn’t used as an excuse for bringing insecurity and spiteful bureaucracy back in (although how long can woefully underperforming/actively dangerous providers carry on with block contracts at huge cost?). Decisions need to be made about uprating the annual amount in line with inflation, whether any underspend is given back (without affecting the next year’s amount) or can be saved for later investment, and how the amount can be revised upwards as people’s needs may change?

If all this sounds too utopian and costly, I would ask you to consider the following:

Local authority returns and personal social services expenditure data suggest that there were 105,305 adults with learning disabilities subject to assessment/care management in 2012/13, at an average annual cost of £3,105 per person per year (my estimate is a total cost per year of £326.9 million). I think these proposals should drastically cut this expenditure.

I’ve been trying to do some estimates on the likely amount of NHS and local authority money being spent directly on adults with learning disabilities. At the moment (and there’s more work for me to do here), I reckon it comes to at least £6 billion per year. Surely much of this could be better spent?

Update: As @milton_damian and @MarkNeary1 have quite rightly pointed out on Twitter, this post assumes that people still have to prove themselves accountable to the state. They rightly argue that:
1) the state should have to prove itself accountable to people, not the other way round
2) Mechanisms of 'accountability' like those proposed above can be and are used as stickes to beat people with
3) Things like DLA, child benefit etc don't require such 'accountability'.

So - scratch that bit of the blogpost above!

Return of the chunky ninjas

This is kind of prompted by a passionate blogpost from @ProfRHastings “Challenging behaviour and learning disability in the UK – the three options” (available here http://profhastings.blogspot.co.uk/2014/10/challenging-behaviour-and-learning.html?m=1 ). This post clearly outlines some points of agreement on the major issues concerning the (mis)treatment of people with challenging behaviour. He calls for a systemic, comprehensive approach to change rather than a piecemeal focus on any particular element of this dysfunctional system, which on its own will be doomed to failure. The three options in the title of the blog are really two; to write another report restating what we already know; or to follow through on well-elaborated ideas from Richard and others for a challenging behaviour academy for England. His third option is really a challenge, to go on if you think you can come up with something better.

As I'm someone who has been on a protracted whingeathon about recent policy concerning inpatient services and support for people with learning disabilities generally, this challenge is well made. So, this blogpost is an attempt to put a bit more flesh on the bones of a post I wrote in June which included a ninja proposal about getting people out of inpatient services and making sure that people don’t go into them (available here http://chrishatton.blogspot.co.uk/2014/06/now-is-winterbourne-of-our-discontent.html ).

As ever, the ideas in the original proposal and their development here are in a real sense collective, as they’re the result of conversations with a lot of people and continuously inspired by the #justiceforLB campaign (all ridiculousness here is mine alone, however). They are also obviously a starting point for more conversations rather than an end-point. Although not ideal (nothing ever is, of course), these ideas also start from where we are (deep, huh?) in terms of current legislation and some major currents in policy.

I think the ideas that follow fit with the “We Have The Right” statement recently produced by people with learning disabilities (available here http://www.changepeople.org/wp-content/uploads/2014/10/CHANGE_Statement_01Oct2014.pdf ).

I also think the ideas here would have a much greater chance of making a real difference to people’s lives if they were underpinned by the #LBBill passing into law (see here for background information to the #LBBill http://lbbill.wordpress.com/2014/09/28/what-is-lb-bill-easy-read-overview/ ).

These ideas are agnostic about a national challenging behaviour academy – such an academy could fit here but isn't assumed – for me the question of the best ways to support people with challenging behaviour would be a matter for the national ninja taskforce.

The original soy skinny ninja proposal was this:

·         Close all of these services to new admissions.

·         CQC to make a clear statement that they will not register new services of this type, and that they will have a process of deregistration of existing services.

·         For all commissioners still funding people in these services, take that funding off them to be administered by a national ninja closure taskforce. This would give the taskforce a starting budget of over half a billion pounds per year (could add in the Winterbourne View programme money too?).

·         Put people with learning disabilities and families really in charge of this ninja taskforce, making decisions about who is appointed to work on the taskforce in which capacities, what the taskforce does, and how it operates.

·         The ninja taskforce works to develop local, individual supports for people moving out of these services – with new people, agencies and supporters rather than with the usual provider suspects (unless the usual provider suspects can come up with something really good that the person and family members want). Obviously this would be person-centred, with the person and family in charge.

·         Funding at its current ATU level moves with the person (howls of protest, two-tier Rolls Royce service, all true, but it would demonstrate what good looks like).

·         The ninja taskforce also help to develop local, individual supports for other people with learning disabilities who might have been sent to these places by commissioners, including strong local structures for self advocacy and family advocacy.

·         The ninja taskforce to help to develop ways of supporting people with learning disabilities in acute distress/crisis for short periods of time.

So, here come the chunky ninjas to describe in a bit more detail what might need to happen to help, focusing a bit more on stopping people going in rather than getting people out (the main focus of the original ninja taskforce proposal above).

Stopping people going in…

1) Start from the beginning. All children identified as having a learning disability will, in theory, have Education and Health Plans (EHCs, which unfortunately sound like a polite Yorkshire retch) which are supposed to support people up to age 25. Strong support and oversight of EHCs are needed if they aren’t to be inconsistently and patchily applied. National standards for the way these plans are produced and reviewed need to be to driven by children/young people and families. It is vital that these plans start from a position of identifying people’s strengths and encouraging/supporting the development of imagined futures (@sarasiobhan’s phrase) with children/young people. In support of this, plans also need to identify children/young people who might be at risk later on of becoming distressed, showing challenging behaviour or breaking the law (including identifying ‘public health’ risk factors like living in poverty and facing adversity early in life).

It is vital that parents and children/young people are the driving force for EHCs, particularly such that parents don’t have to highlight ‘problems’ for them to get good support. Children/young people and families need to have control over the aims of the EHC plan, there needs to be transparency and predictability about the resources underpinning each plan, and they need to control how the plan is to be delivered (with a minimum of bureaucracy).

Local areas should be required to produce regularly updated, publically available (and easy to find) information about the number and needs of children and young people up to the age of 25 in their area, and what is being done to help children/young people and their families to get the support they want.

It is also important that local areas have a really good handle on children/young people in residential special schools, whether in or out of area. A parallel process to inpatient service closure for adults (and the prevention of admissions to residential special schools) should be instituted for residential special schools.

An additional source of information and support should be annual health checks for people with learning disabilities, which are being extended to include young people with learning disabilities from the age of 14. Current financial incentives for GPs have only resulted in just over half of ‘eligible’ adults with learning disabilities receiving annual health checks. Strengthening the requirement for GP practices to register people with learning disabilities and to conduct annual health checks is needed. It’s also vital that health checks for young people with learning disabilities cover the important issues for this group, including potential signs of distress – these should be closely linked to and work with EHCs. National standards for the way that health checks are done need to be to be driven by children/young people and families, with professionals in support.

2) Don’t write anyone off. Each local area needs to have good, regularly updated and publically available information about:

a) Adults who already show behaviour that challenges and/or mental health issues.  

b) Adults who experience crises – they may not be known to health or social services and even if known, may not meet social care eligibility criteria until the crisis happens. This is obviously less easy to predict, but local areas need to have some idea of how many people in crisis are likely to emerge in any given timespan (and to offer this group of people low level support in advance of a crisis, including advance planning for if/when likely crises/changes in people’s circumstances occur).

c) There is the group of offenders who have been diverted from prison, and there are also prisoners with learning disabilities. Local areas need to have good information on these groups, including what their offences are, what the prison term would have been/is and what level of security is specified for them at the moment.

3) I think Integrated Personalised Commissioning (as far as I understand it, the ‘official’ NHS England term for personal health budgets, but they should draw in social care funding too) are important here, especially as NHS England are pushing this approach generally. Where will the money come from? Well, if personalised commissioning is taken seriously then existing boundaries between commissioners (NHS specialised commissioning, local Clinical Commissioning Groups (CCGs), and local authorities) need to be dissolved, as these boundaries  automatically create incentives for different commissioners to try and shunt costs on to someone else. Could funding from these sources be pooled, then commissioners would be much more about helping people (individually and collectively) to get the support they want rather than commissioning services that they then have to squish people into?

At a local level, it needs to be possible for people/families/allies/advocates to come together to specify what support they want and work with potential providers of that support so they can get that support. We need to avoid over-complicated tendering/procurement processes and block contracting so that people can get what they really want/need (potentially from new or very small organisations) rather than necessarily what existing big support providers already do (although some people will want what service providers already do, which is fine). In some places it might make sense for some support services for people across different local areas to band together?

Rather than having block contracts for big service providers, why not have guaranteed 10-year amounts for people’s IPCs (people could return some of their IPC if they thought it was too much, and there would be the option for the amount of the IPC to increase if necessary). The money always follows the person.

For individuals, Circles of Support or other ways of supporting personal budgets could be used as models for supporting people and families to make good use of their IPCs. Good independent advocacy and peer advocacy should also be available to all people who want it. There can be multiple vehicles for delivering IPCs (including but not confined to direct payments, broker managed budgets, provider managed budgets etc). The person/family/allies need to be in control of which vehicle suits them (and retain the option to change this), the whole process needs to be easy, transparent, reliable and fair, and require a minimum of paperwork to manage.

IPCs should include advance planning for potential crisis situations – what does the person/family/allies want to happen in those circumstances and how is that support going to be available?

People/families/allies need to have support/expertise/resources developed regionally/nationally to draw on if they wish to. The national ninja taskforce is definitely needed (led by people with learning disabilities/families/allies) to direct operations and keep everyone honest.

4) I think annual health checks (and more specialist health support) have a crucial role to play for everyone, in checking for health conditions that might be associated with discomfort, pain, and behaviours potentially viewed as challenging or indicative of a mental health problem and resolving the health condition. The issue of medication (particularly multiple medicines, particularly particularly psychotropic medication, and particularly particularly particularly antipsychotic medication) is obviously crucial as well. There needs to be a strong, enforceable way to ensure that medications are reduced and kept to a minimum.

5) Can anyone issue a blanket order preventing any new admissions to particular types of inpatient services? I suspect not, in which case there need to be strong contingencies such that it’s much harder for commissioners to get someone into an inpatient unit than to provide decent community-based support. The #LBBill is particularly important here. It’s also vital that people who have committed a crime but been diverted from prison, and residential special schools, are not left out (or more to the point, left in).

6) NHS England should impose tough, nationally specified contracting standards on existing commissioners (specialised commissioners and CCGs) setting out in very clear terms what is required in terms of commissioning places in inpatient services. This might include the operation of the #LBBill before a placement is decided, a complete health assessment and time-limited discharge plan set on arrival. And how about a person being placed in an inpatient unit triggering an IPC, with the amount set at a minimum at the inpatient service weekly fee and to follow the person once discharged?

7) The CQC making a clear statement about models of inpatient service they will not register, with plans for new inpatient services having to be submitted to CQC in advance to them for adjudication.

A quick word on getting people out…

I think pretty much everyone who is currently in inpatient services (with the possible exception of the 73ish people with learning disabilities in high secure services) could also plug into the way of doing things outlined above, with the money for the IPC following them (from NHS specialised commissioners and CCGs mainly). Even if people require a degree of security, there is no reason why a circle of support and an IPC would not apply to them, presumably with one aim among others of needing less security over time. As people move out of inpatient services, they could then plug into and become part of the local networks described above.

Where’s the money?

Changing the way any system operates always costs money, at least while the system changes. Although this is obviously not a fully costed proposal, there are some sources of potential savings that should arise from this proposal to help support the proposed changes, including:

1) The cost of specialist inpatient services for adults with learning disabilities, currently running at over half a billion pounds per year.

2) The cost of residential special schools for children and young people with learning disabilities. Although recent data are unavailable, there are likely to be at least 1,000 children and young people in residential special schools in England. If these are funded at similar levels of assessment and treatment units, closing residential special schools would save at least £150 million per year.

3) There are around 30,000 adults with learning disabilities in England in some form of residential or nursing care funded by social care, at a cost of approximately £2.1 billion per year. Assuming that half of these residential placements would not be the preferred option for people with learning disabilities would in itself provide £1 billion a year for re-investment.

4) There are some other ‘big ticket’ items of social services expenditure on services for people with learning disabilities that some people with learning disabilities may not choose to keep using, such as day centres (current cost over £700 million per year).

5) Finally, there are a number of smaller areas of potential savings (for example, less prescription of antipsychotic medication) that are all relevant when evaluating the cost implications of these ideas.

In total, I think £2 billion per year wouldn’t be a ridiculous figure in terms of funding for reinvestment in a new way of doing things, not to mention ongoing savings from putting in decent, early support so that fewer people end up in crisis in the first place.

Personally, I worry that for the short-term injection of cash, social investment models ask too high a price in terms of the amount of money that would be sucked out of supporting people with learning disabilities into the pockets of social investors. If an investment model of this type is needed (and I think new buildings are less important than building good support from good people), what about some kind of national or regional state investment bank, with guaranteed low rates of interest and the returns being used for the bank to make further investments in supporting people with learning disabilities?

Chunky ninjas

So, the ninjas have put on a little weight – what do you think?