LPT science – getting it right and getting it wrong

A comment on: Kadhim-Saleh A, Maganti H, Ghert M, Singh S, Farrokhyar F. Is low-level laser therapy in relieving neck pain effective? Systematic review and meta-analysis. Rheumatol Int. 2013 Oct;33(10):2493-501.

By Jan Tunér

 

In a recent paper in The Lancet (1) the authors note that too many scientific papers are flawed by misinterpretation of previous research in the field, of not reporting the full parameters and of boosting the results of their own study. The object of the LaserAnnals is to scrutinize the LPT literature and to try to squeeze the truth out of seemingly negative studies. To us, the criticism in the Lancet paper comes as no surprise, unfortunately. The reader of this blog will already have noticed that “science” is a minefield with a considerably high number of casualties.

The average reader of an LPT scientific paper will have difficulties interpreting the trustworthiness of a study. And to evaluate a systematic review or Meta analysis, the reader needs to be very qualified in everything from the medical aspects to the statistical ones. And all the tables and statistical analyses appear to be the guarantee for a solid job. Unfortunately, this is not the general rule; even very qualified researchers do put their feet on a mine every now and then. Here is a casualty report.

Kadhim-Saleh (2) recently published a systematic review and Meta-analysis on the effect of LPT in relieving neck pain. The conclusion in the paper is “The benefit seen in the use of LLLT, although statistically significant, does not constitute the threshold of minimally important clinical difference”. Here is the abstract of the paper:

 

The aim of this study is to determine the efficacy of low-level laser therapy (LLLT) in reducing acute and chronic neck pain as measured by the visual analog scale (VAS). A systematic search of nine electronic databases was conducted to identify original articles. For study selection, two reviewers independently assessed titles, abstracts, and full text for eligibility. Methodological quality was assessed using the Detsky scale. Data were analyzed using random-effects model in the presence of heterogeneity and fixed-effect model in its absence. Heterogeneity was assessed using Cochran’s Q statistic and quantifying I (2). Risk ratios (RR) with 95 % confidence intervals (CI) were reported. Eight randomized controlled trials involving 443 patients met the strict inclusion criteria. Inter-rater reliability for study selection was 92.8 % (95 % CIs 80.9-100 %) and for methodological quality assessment was 83.9 % (95 % CIs 19.4-96.8 %). Five trials included patients with cervical myofascial pain syndrome (CMPS), and three trials included different patient populations. A meta-analysis of five CMPS trials revealed a mean improvement of VAS score of 10.54 with LLLT (95 % CI 0.37-20.71; Heterogeneity I (2) = 65 %, P = 0.02). This systematic review provides inconclusive evidence because of significant between-study heterogeneity and potential risk of bias. The benefit seen in the use of LLLT, although statistically significant, does not constitute the threshold of minimally important clinical difference.

 

This seems to be a qualified and thorough analysis and thereby important for the potential use of LPT for this kind of indication. And certainly it will be quoted as the gospel truth in many forthcoming papers. But what is the actual validity of the analysis? It takes a qualified team of experienced researchers (3) to see through the claims of the paper by Kadhim-Saleh et al. Such a criticism is appearing as a “Letter to the Editor” in the same journal. It is heavy reading, but well worth a try, since it pinpoints the possibilities of casualties in the scientific minefield.

 

We will deal with the most serious issue first. Kadhim-Saleh et al. excluded a study from one of our investigating team stating that “We further excluded Chow et al. [3] because it most likely included the same cohort as Chow et al. [4]”. This serious error would not have occurred if the reviewers were diligent in their processes. The published trial reports clearly stated that one cohort was recruited in 1998 and 1999, and the other cohort recruited in 2002–2003. Rather than contacting the trial authors to resolve uncertainty about the cohorts, which is normal practice for systematic reviewers Kadhim-Saleh et al. decided to base their decision on guess work, clearly not stringent reviewing practice. The hopefully unintentional insinuation that Chow and colleagues duplicated a cohort and included these duplicates as if they were separate cohorts in a subsequent meta-analysis published in the Lancet should be retracted. Kadhim-Saleh et al. criticize our previous meta-analysis because “… investigators included trials that used different and more subjective tools for assessing the primary pain outcome measure. p2494”. In fact we included studies that used several well-recognized and validated scales for pain assessment. This included calculating weighted mean difference for continuous data from visual analogues scale (VAS) scores for pain intensity, relative risk for dichotomized data for global improvement, and standardized mean difference to combine different validated scales of disability including the Neck Disability Index, Neck Disability Scale and the Northwick Park Questionnaire. This enabled us to conduct a comprehensive analysis of the available evidence and we believe that our finding that the results were consistent across the different measurement tools strengthened the robustness of our conclusion. Kadhim-Saleh et al. used a single outcome measure at a single time-point as their only outcome measure of success. We are concerned

In essence, we challenge the reviewers claim that they used “A very strict study selection criterion … p2494” that this approach does not provide a full picture of the available evidence. For example, Kadhim-Saleh et al. included a new study by Konstantinovic et al. [5] that failed to detect a statistically significance difference between LLLT and placebo based on the outcome measure for pain on VAS. However, Konstantinovic et al. measured seven other outcomes and all were in favour of LLLT. Moreover, the study population was acute radiculopathy, which is a specific type of neck pain, and this would have increased heterogeneity associated with differences in study populations. There are dangers in separating trials in diagnostic categories that cannot be separated because they are overlapping, not exclusive, and their objective diagnostic tests are lacking validity [6]. It was for this reason that we limited conditions to non-specific neck pain in our review. Kadhim-Saleh et al. criticized our meta-analysis for having heterogeneity despite I2 appearing to be larger in their analysis (I2 = 94 % compared with our analysis I2 = 91 %). This suggests to us that the “stringent” criteria used by Kadhim-Saleh et al. failed to resolve high heterogeneity. Heterogeneity is common in studies on pain, so reviewers should be explicit about how they manage it. We included a 650-word paragraph in our Results section describing a sensitivity analyses that found that most heterogeneity was attributed to interventions. This view has been supported by others [7, 8]. Kadhim-Saleh et al. appeared to give no serious consideration to the appropriateness of LLLT technique including dosage a priori in selection criteria or analysis protocol. An example of how LLLT dose explains heterogeneity is provided by examining the two acute group trials included in our review [9, 10]. The negative trial using LLLT on acupuncture points by Aigner et al. [10] was under-dosed (632 nm wavelength, 0.075 J) with an ineffective dose of only 0.02 % when compared with the positive trial by Soriano et al. [9] (904 nm wavelength, 4 J). The additional acute radiculopathy trial by Konstantinovic et al. included in the review by Kadhim-Saleh et al. had similar doses and wavelength (904 nm wavelength, 2 J) to the positive trial by Soriano et al., and both were complying with the dosage recommendations from World Association for Laser Therapy [11]. By conducting a meta-analysis that excludes the trial by Aigner et al. and substituting it with the new acute radiculopathy trial by Konstantinovic et al., we have found that heterogeneity disappears completely (from I2 = 91 % to I2 = 0 %). This results in a significant and clinically relevant RR for global improvement at 2.63 (95 % CI 1.73, 4.01). The authors also forward an unsubstantiated claim about a re-analysis of our meta-analysis by Shiri and Viikari-Juntura [12] “After applying a random-effects model Shiri and Viikari-Juntura found no significant difference between laser-treated and placebo-treated groups in pain reduction”. This is simply untrue, as Shiri and Viika-Junturi confirmed that pain reduction on VAS with a weighted mean difference of 19.41 (95 % CI 9.67, 29.15) in a random-effects model. The inaccuracies in our original analysis brought to our attention by Shiri and Viikari-Juntura weakened slightly the size of effect on recalculation, yet the overall result remained that LLLT gave significant and clinically relevant relief for 6 out of 8 outcomes and lasting up to 22 weeks. Finally, we do not agree that exclusion of non-English trial reports, without reason, is consistent with robust study selection criterion although Kadhim-Saleh et al. concede that this was a methodological limitation in their Discussion. They cited meta-analyses published over 20 years ago to demonstrate the consistency of their claim with previous reviews that found no effect from LLLT despite 80–90 % of RCTs on LLLT being published after these citations. The rate of publication of RCTs in recent years means that the survival period for systematic reviews is typically less than 5 years [13]. In addition, they used literature published between 14 and 30 years ago to support descriptions of LLLT mechanisms. In conclusion, we are concerned about the methodology implemented in the review by Kadhim-Saleh et al. and believe that it may have led to a misleading conclusion. The information provided in the published report does not support their claim that they used “very stringent” review methodology. We also ask readers to reconsider the claim of inconclusive evidence for LLLT in chronic neck pain presented by Kadhim-Saleh et al. Available evidence could, be interpreted as LLLT gives significant global improvement in acute neck pain with and without radiculopathy. We stand by our original findings that LLLT gives clinically relevant neck pain relief and disability improvement after treatment and possibly follow-ups up to 5 months.

(References not included here)

 

This Letter to the Editor corrects the mistakes in the original paper. However, the impact of such a letter is small. The abstract of the original paper appears on PubMed while the comments appear without an abstract. It is therefore suggested that the “PubMed commons” option be used more frequently in the future. It makes the comment available faster and readily available to anyone reading the original study abstract.

 

(1)  Malcolm R Macleod, Susan Michie, Ian Roberts, Ulrich Dirnagl, Iain Chalmers, John P A Ioannidis, Rustam Al-Shahi Salman, An-Wen Chan, Paul Glasziou. Biomedical research: increasing value, reducing waste. The Lancet. Jan 11, 2014; Volume 383, Number 9912.

(2)  Kadhim-Saleh A, Maganti H, Ghert M, Singh S, Farrokhyar F. Is low-level laser therapy in relieving neck pain effective? Systematic review and meta-analysis. Rheumatol Int. 2013; 33 (10): 2493-2501.

(3)  Bjordal JM, Chow RT, Lopes-Martins RA, Johnson MI. Methodological shortcomings make conclusion highly sensitive to relevant changes in review protocol. Rheumatol Int. 2014 Jan 9. [Epub ahead of print]