Acupuncture for infantile colic – misdirection in the media or over-reaction from a sceptic blogger?

This blog was first published on 26th January 2017 on https://blogs.bmj.com/aim/. At the time I was in Cape Town on holiday, trying to get a rapid response published to the NG59 summary in the BMJ. It was critical of NICE, and I was negotiating over content with a legal expert from BMJ! The response took three weeks to go up, by which time it was too late to be noticed. In the meantime I created a bit of a storm with this blog, and my use of the term ‘old sceptic blogger’ in the title. This is (mostly) the version edited by BMJ.

Today-programme-BBC-R4

So there has been a big response to this paper press released by BMJ on behalf of the journal Acupuncture in Medicine. The response has been influenced by the usual characters – retired professors who are professional bloggers and vocal critics of anything in the realm of complementary medicine. They thrive on flexing their EBM muscles for a baying mob of fellow sceptics (see my ‘stereotypical mental image’ here). Their target in this instant is a relatively small trial on acupuncture for infantile colic.[1] Deserving of being press released by virtue of being the largest to date in the field, but by no means because it gave a definitive answer to the question of the efficacy of acupuncture in the condition. We need to wait for an SR where the data from the 4 trials to date can be combined.

On this occasion I had the pleasure of joining a short segment on the Today programme on BBC Radio 4 led by John Humphreys. My protagonist was David Colquhoun, who spent his short air-time complaining that the journal was even allowed to be published in the first place. Why would BBC Radio 4 invite a retired basic scientist and professional sceptic to be interviewed alongside one of the journal editors – a clinician with expertise in acupuncture (WMA)?

At no point was it made manifest that only one of us had ever been in a position to try to help parents with a baby that cries excessively. 

So what about the research itself? I have already said that the trial was not definitive, but it was not a bad trial. It suffered from under-recruiting, which meant that it was underpowered in terms of the statistical analysis. But it was prospectively registered, had ethical approval and the protocol was published. Primary and secondary outcomes were clearly defined, and the only change from the published protocol was to combine the two acupuncture groups in an attempt to improve the statistical power because of under recruitment. The fact that this decision was made after the trial had begun means that the results would have to be considered speculative. For this reason the editors of Acupuncture in Medicine insisted on alteration of the language in which the conclusions were framed to reflect this level of uncertainty.

David Colquhoun has focussed on multiple statistical testing and p values. These are important considerations, and we could have insisted on more clarity in the paper. P values are a guide and the 0.05 level commonly adopted must be interpreted appropriately in the circumstances. In this paper there are no definitive conclusions, so the p values recorded are there to guide future hypothesis generation and trial design. There were over 50 p values reported in this paper, so by chance alone you must expect some to be below 0.05. If one is to claim statistical significance of an outcome at the 0.05 level, ie a 1:20 likelihood of the event happening by chance alone, you can only perform the test once. If you perform the test twice you must reduce the p value to 0.025 if you want to claim statistical significance of one or other of the tests. So now we must come to the predefined outcomes. They were clearly stated, and the results of these are the only ones relevant to the conclusions of the paper. The primary outcome was the relative reduction in total crying time (TC) at 2 weeks. There were two significance tests at this point for relative TC. For a statistically significant result, the p values would need to be less than or equal to 0.025 – neither was this low, hence my comment on the Radio 4 Today programme that this was technically a negative trial (more correctly ‘not a positive trial’ – it failed to disprove the null hypothesis ie that the samples were drawn from the same population and the acupuncture intervention did not change the population treated). Finally to the secondary outcome – this was the number of infants in each group who continued to fulfil the criteria for colic at the end of each intervention week. There were four tests of significance so we need to divide 0.05 by 4 to maintain the 1:20 chance of a random event ie only draw conclusions regarding statistical significance if any of the tests resulted in a p value at or below 0.0125. Two of the 4 tests were below this figure, so we say that the result is unlikely to have been chance alone in this case. With hindsight it might have been good to include this explanation in the paper itself, but as editors we must constantly balance how much we push authors to adjust their papers, and in this case the editor focussed on reducing the conclusions to being speculative rather than definitive. A significant result in a secondary outcome leads to a speculative conclusion that acupuncture ‘may’ be an effective treatment option… but further research will be needed etc…

Now a final word on the 3000 plus acupuncture trials that David Colquhoun mentions. His point is that there is no consistent evidence for acupuncture after over 3000 RCTs, so it clearly doesn’t work. He first quoted this figure in an editorial after discussing the largest, most statistically reliable meta-analysis to date – the Vickers et al IPDM.[2] He admits that there is a small effect of acupuncture over sham, but follows the standard EBM mantra that it is too small to be clinically meaningful without ever considering the possibility that sham (gentle acupuncture plus context of acupuncture) can have clinically relevant effects when compared with conventional treatments. Perhaps now the best example of this is a network meta-analysis (NMA) using individual patient data (IPD), which clearly demonstrates benefits of sham acupuncture over usual care (a variety of best standard or usual care) in terms of health-related quality of life (HRQoL).[3]

Key to abbreviations

  • BMJ – British Medical Journal (company)
  • EBM – evidence-based medicine
  • HRQoL – health-related quality of life
  • IDP – individual patient data
  • IDPM – individual patient data meta-analysis
  • MCID – minimal clinically important difference
  • NMA – network meta-analysis
  • SR – systematic review
  • VAS – visual analogue scale (usually a 100mm line)

References

  1. Landgren K, Hallström I. Effect of minimal acupuncture for infantile colic: a multicentre, three-armed, single-blind, randomised controlled trial (ACU-COL). Acupunct Med 2017: acupmed-2016-011208. doi:10.1136/acupmed-2016-011208
  2. Vickers AJ, Cronin AM, Maschino AC, et al. Acupuncture for chronic pain: individual patient data meta-analysis. Arch Intern Med 2012;172:1444–53. doi:10.1001/archinternmed.2012.3654
  3. Saramago P, Woods B, Weatherly H, et al. Methods for network meta-analysis of continuous outcomes using individual patient data: a case study in acupuncture for chronic pain. BMC Med Res Methodol 2016;16:131. doi:10.1186/s12874-016-0224-1

Declaration of interests MC

Trust Me, I’m an acupuncture expert – but I have never actually had it or used it…

This blog was first published on 4th September 2016 on https://blogs.bmj.com/aim/

MC-trust-me-2016-1024x576

On Thursday 1st September the first episode of series five of Trust Me I’m A Doctor aired on BBC2. I was keen to see how acupuncture was treated after spending a day engaged in trying to demonstrate a change in pressure pain threshold in the lead presenter about a month previously. The experiment went relatively well, and Michael’s pressure pain thresholds doubled from before to after the experiment. Sham acupuncture involved the use of ‘non-penetrating’ retractable needles. It was the first time I had used these in earnest and they succeeded in masking the subject – Michael could not tell which of the interventions involved real acupuncture. I did note that the sham needles could inadvertently penetrate the skin, particularly if the retractable shaft had an overly stiff sliding action. Michael found the sham needling created quite a strong sensation, and I had to work quite hard to create as strong a sensation with the real acupuncture – I found this very interesting as someone who has used acupuncture therapeutically for over 20 years, but never actually tried to perform sham acupuncture in a ‘trial’. Experienced acupuncture practitioners who then have to perform sham interventions have often remarked to me that the sham techniques do far more than they expect in terms of sensation, and in the case of the ‘so called’ non-penetrating needles, how often they cause bleeding.

Madsen et al got away with pooling data from acute and chronic pain, from surgical pain to headache to arthritis…

So that brings us to the ‘acupuncture expert’, who, according to ‘Trust Me’, “…has spent much of his career studying the effect of acupuncture.” If you go to PubMed and insert in the search box: Hróbjartsson A [au] AND acup*; you will only get 4 papers, and only one of them will have acupuncture in the title. That is the infamous BMJ systematic review (Madsen et al BMJ 2009)[1] that got away with pooling data from trials of acute and chronic pain, from surgical pain to headache to arthritis. Yes I did say pooling. The clinical heterogeneity in this review was simply breathtaking, but I guess that the relevant BMJ editors were eclipsed by the home address of the authors – the esteemed Nordic Cochrane Centre. But this was not a review performed within the remit of the Cochrane Collaboration. As a Cochrane author I know the rigors of the process very well, and I can assure readers that Madsen et al would never pass muster in such an arena. Yet the authors of the review used this address, perhaps to their advantage in securing a prominent publication.

So we have an expert with one highly controversial review paper on acupuncture to his name. An expert who has never received acupuncture treatment let alone used it. An expert who thinks we do not know how it works, despite over 60 years of laboratory data investigating mechanisms from endogenous opioids to adenosine release.[2] Dare I say, a medical expert who has never touched a patient therapeutically?

So yes, I have to admit I am disappointed with the superficial way the subject was covered, and the lack of acknowledgement of the challenges of performing blinded trials of acupuncture. Challenges that are eminently illustrated by Haake et al (2007)[3] – the biggest ever sham controlled trial of acupuncture in low back pain, with over 1000 patients. In this trial, sham acupuncture performed twice as well as rather intensive German guideline-based conventional care. Can our acupuncture expert really propose that this is simply a placebo response?

References

  1. Madsen MV, Gøtzsche PC, Hróbjartsson A. Acupuncture treatment for pain: systematic review of randomised clinical trials with acupuncture, placebo acupuncture, and no acupuncture groups. BMJ 2009;338:a3115. doi:10.1136/bmj.a3115
  2. Filshie J, White A, Cummings M. Medical Acupuncture – A Western Scientific Approach. 2nd ed. Elsevier 2016.
  3. Haake M, Müller H-H, Schade-Brittinger C, et al. German Acupuncture Trials (GERAC) for chronic low back pain: randomized, multicenter, blinded, parallel-group trial with 3 groups. Arch Intern Med 2007;167:1892–8. doi:10.1001/archinte.167.17.1892

Declaration of interests MC