Do sham devices impede real acupuncture?

Stimulated by Kim et al 2019.[1]

I have been waiting for this network meta-analysis to be published since reviewing it 18 months ago. It introduces data to support an idea that I have had in my head for some time but never thought to actually try to quantify. This paper does that by comparing the effect of acupuncture when using sham devices with the effect using acupuncture needles normally. This comparison has never been made directly in a clinical trial of course, and would be problematic in terms of blinding, since the sham devices are difficult to ignore or conceal. So, the comparison is made indirectly within a network meta-analysis including sham controlled trials where the sham is either non-penetrating (requiring a device to hold the needle) or penetrating (shallow needling for example).

…bleeding with non-penetrating needles!

Helen Elden (personal communication)

I first became aware of the potential problem of the non-penetrating sham devices when Helen Elden remarked to me at a meeting that she was surprised how often she saw bleeding with the non-penetrating needles in her sham controlled trial of acupuncture for pelvic girdle pain in pregnancy.[2,3] Many years later I was in the position of having to use these devices in earnest for a TV programme in which the presenter wanted to be blinded to the real or sham intervention (see my previous blog – Trust me…). I struggled more in performing the real needling through the sham devices (the Park sham device) because at first I inadvertently placed the needles through the stick pad that held the device upright on the skin. This meant that they were very hard to insert and manipulate. Consequently, less was felt in the real acupuncture group until I made an excuse and reinserted the needles through the tiny hole in the middle of the sticky pad. I was particularly careful when using the non-penetrating needles, but when chatting with the director later I easily pushed one through his skin by accident!

…the real needles got stuck in the sticky pads!

MC (frustrating personal TV experience)

Anyway, this is really about the effect in the real acupuncture groups rather than the sham groups. It is particularly pertinent since the update of the Acupuncture Trialist Collaboration Individual Patient Data Meta-analysis (the subject of a previous blog) found that the difference between real and sham was greater when non-penetrating devices were used.[4] I was concerned that researchers would be encouraged to use these devices without considering how they might degrade the effect in the real acupuncture groups. This paper adds some weight to the argument against the use of non-penetrating sham devices.

They chose to look at acupuncture trials in hot flushes, and I have not asked why this was the focus, despite meeting one of the authors in Munich last year, another in Madrid this year, and having email correspondence with a third about a meeting in Oslo next year. I guess I should email the first author to complete my contact with the full list of authors!

Briefly, network meta-analysis is a method for combining data from multiple two-way comparisons of interventions (individual RCTs or multiple RCTs combined in meta-analysis) so that both direct and indirect comparisons between interventions can be performed (see my previous blog – Quality sham). This network consisted of eight RCTs, so this is a relatively small network. Figure 2 (below) from the paper shows the network. The blue dots are called nodes, and represent different interventions: A for acupuncture; S for sham; W for waiting list; 1 for non-penetrating sham device; 2 for penetrating. The thickness of the lines represents the number of studies directly comparing each intervention.

The comparison of interest is A1 with A2, where there is no line, so this is called an indirect comparison. The data from each trial was extracted and converted to standardised mean differences (SMDs) as the outcome measures used in the trials were not directly comparable.  These figures were then put into a fancy statistical software package to perform the network meta-analysis (NMA).

Table 1 (below) shows the results of the NMA. The interventions form a diagonal line across the table from top left to bottom right. The direct comparisons are above and right and fewer in number. The indirect comparisons are below and left. Choose a treatment and read the direct comparisons from left to right and the indirect from right to left to compare it with a second treatment. If the SMD is positive it means the first treatment selected is superior, and if negative it means the second treatment selected is superior.

This gives us the result that whilst both A1 and A2 are superior to their respective shams (S1 and S2 respectively), A2 is markedly superior to A1 and S2 markedly superior to S1. So acupuncture performed for hot flushes without a sham device is much more effective than acupuncture for hot flushes performed through a sham device. Also of note, shallow needling (the sham technique used in trials that do not use the sham devices – S2) is also markedly superior the real acupuncture performed through a sham device (A1). The SMDs for these comparisons run to around 6 to 7, which is an enormous effect size – a large SMD or effect size is said to be 0.8 or above. I had never seen such a large effect in any comparison of acupuncture with anything else before. Then I noticed the 8.61 in the upper right of Table 1. This was a direct comparison of acupuncture and waiting list, with a huge effect size. So I checked the original paper, and it seems to be sound, with a relatively large number of treatment sessions, and quite narrow SDs (standard deviations).[5] Since effect size as SMD is calculated by dividing the mean difference between two groups by the SD of the difference, a narrow SD will inflate the effect size.

Anyway, the clear indication from this paper is that sham devices may degrade the effect of real acupuncture in trials of hot flushes. I am now wondering whether or not a similar effect will be seen in the chronic pain data. I can sense an enormous NMA on the horizon.

References

1         Kim T-H, Lee MS, Alraek T, et al. Acupuncture in sham device controlled trials may not be as effective as acupuncture in the real world: a preliminary network meta-analysis of studies of acupuncture for hot flashes in menopausal women. Acupunct Med 2019;:acupmed2018011671. doi:10.1136/acupmed-2018-011671

2         Elden H, Ladfors L, Olsen MF, et al. Effects of acupuncture and stabilising exercises as adjunct to standard treatment in pregnant women with pelvic girdle pain: randomised single blind controlled trial. BMJ 2005;330:761. doi:10.1136/bmj.38397.507014.E0

3         Elden H, Fagevik-Olsen M, Ostgaard H-C, et al. Acupuncture as an adjunct to standard treatment for pelvic girdle pain in pregnant women: randomised double-blinded controlled trial comparing acupuncture with non-penetrating sham acupuncture. BJOG An Int J Obstet Gynaecol 2008;115:1655–68. doi:10.1111/j.1471-0528.2008.01904.x

4         Vickers AJ, Vertosick EA, Lewith G, et al. Acupuncture for Chronic Pain: Update of an Individual Patient Data Meta-Analysis. J Pain 2017;19:455–74. doi:10.1016/j.jpain.2017.11.005

5         Avis NE, Coeytaux RR, Isom S, et al. Acupuncture in Menopause (AIM) study: a pragmatic, randomized controlled trial. Menopause 2016;23:626–37. doi:10.1097/GME.0000000000000597

---

Declaration of interests MC