Evidence by excellence or by evidence?

By Jan Tunér

In the “old days” the medical treatment recommendations were based upon opinions from leading medical professionals. These recommendations were generally based upon experience, which is not at all too bad, but not always upon evidence. The situation was changed when the Cochrane Collaboration was launched. Suddenly a strict analysis of the available scientific literature was the basis of treatment recommendations. And in addition, new evaluators entered the system and many previous experts were phased out.

Certainly, the Cochrane system brought about a much better basis for the evaluation of medical therapies. But no system is perfect. For instance, how are the experts in the many groups selected? Is there a circulation of members in the groups or can there be a tendency of “Kamaraderei” – closed fellowships? My own experience is limited to the small number of Cochrane reports in PBM there is, but I am not impressed. The first Cochrane report on PBM was on pressure wound healing. The evaluators found that there was no scientific evidence for the use of PBM in wound healing (1). They were right, but based upon incorrect information. The number of papers was low and did not report about the used laser parameters in any detail. And some of the used studies were not even laser studies, but LED studies. The evaluators apparently had very little knowledge about PBM. What they could not know was that two of the studies analysed were full of flaws or even just fabricated (2,3). I contacted the authors but didn’t receive any answers. So I contacted a high ranked Cochrane member and he informed me that if a Cochrane group failed to answer to or to publish received criticism, the chairman of the group should be sacked. Using this information, I managed to have my criticism published on the online Cochrane system, but it took about two years. And there was no opening for a fruitful discussion. I published my criticism in the journal Laser Therapy in 1999 (4).

The next problem was the Cochrane group analyzing PBM for osteoarthritis and rheumatoid arthritis (5,6). Here too, the authors did not include details of the laser parameters, seemingly unaware of their importance or the importance of papers lacking such information. No reply was received from the chairman of this group and things were not improved by the same person publishing a clinical study of her own (7), confirming the lack of understanding of what she was supposed to analyse. That study (7) was criticized in a Letter to the editor (8).

So much for my own experience of the Cochrane systems and its PBM evaluators. Fortunately, I am not alone. Jan Bjordal, professor at the University of Bergen, Norway is a prominent expert of PBM and on scientific methodology. Bjordal (9) has suggested that the study (5) is biased. Professor Bjordal has performed many analyses of LLLT evaluations and also found many questionable conclusions. Here is an example:

Bjordal JM, Lopes-Martins RA, Klovning A. Is quality control of Cochrane reviews in controversial areas sufficient? J Altern Complement Med. 2006 Mar;12(2):181-3.

Some conclusions from the Cochrane Database of Systematic Reviews (CDSR) seem volatile in areas of controversy and have recently changed conclusions. With this perspective in mind we decided to test the validity and sensitivity of negative conclusions in a systematic review of low-level laser therapy (LLLT) for osteoarthritis (OA) from the Cochrane Library was valid and robust. Validity was tested against a 9-item checklist for systematic reviews. Review selections were analyzed for possible discrepancies between trial and review reports and omissions of relevant trials and data. Data from discrepancies and omissions were then entered into sensitivity and subgroup analyses.

Continuous and dichotomous data for pain. Patients with osteoarthritis:

Only clinicians who had performed LLLT trials, and had negative results were invited into the review group. The review was oblivious to findings published after 1993 about physiologic mechanisms and dose response patterns for LLLT. We found 18 questionable selections that favored a negative review conclusion in 17 of 18 cases. These were largely omissions of relevant positive data and selective inclusion of negative data from trials with small, ineffective doses. When existing and omitted data from relevant trials were combined, results changed from negative to significantly positive for continuous and categorical data. Subgrouping trials by location and recommended doses taken from current guidelines, revealed a highly significant effect of LLLT for treating knee OA. The results of this CDSR was not robust and seems to be colored by questionable selections or omissions. For alternative pain therapies, it should be considered if lack of expertise on therapy mechanisms, or conflicts of interests with competing pain drug manufacturers, may bias conclusions. Review groups should recruit a balanced mix of current views and expertise and expand the use of sensitivity analyses to improve quality of CDSRs in areas of controversy.

Further to that, the Bjordal group has published these critical analyses:

Bjordal JM, Demmink JH. Review of tennis elbow was biased. BMJ. 2004 Apr 10;328(7444):897-898.

Bjordal JM. Evidence-Based Medicine Turned Upside Down. Photomed Laser Surg. 2015 Aug;33(8):391-392.

We can conclude that the methodology offered by modern science are indispensable but not free of flaw and we need to be vigilant. A brilliant career in one part of medicine doesn’t necessarily bring about competence in other parts. A fine title is no guarantee. The aim of the LaserAnnals is precisely to underline this fact.


  1. Flemming K, Cullum N: Laser Therapy for venous leg ulcers (Cochrane review). In: The Cochrane Library. 4, 2000.


  1. Lundeberg T, Malm M. Low power HeNe laser treatment of venous leg ulcers. Ann Plast Surg. 1991; 27 (6): 537-539.


  1. Malm M, Lundeberg T. Effect of low power gallium arsenide laser on healing of venous ulcers. Scand J Plast Reconstr Hand Surg. 1991; 25 (3): 249-251.


  1. Tunér J. The Cochrane analyses - can they be improved? Laser Therapy. 1999; 11 (3):


  1. Brosseau L, Welch V, Wells G et al. Low level laser therapy (classes I, II and III) for treating rheumatoid arthritis. In: The Cochrane Library. Issue 4, 2000. Oxford: Update Software.


  1. Brosseau L, Welch V, Wells G et al. Low level laser therapy (classes I, II and III) for treating osteoarthritis. In: The Cochrane Library. Issue 4, 2000. Oxford: Update Software. Withdrawn 2007.


  1. Brosseau L, Wells G, Marchand S, Gaboury I et al. Randomized controlled trial on low level laser therapy (LLLT) in the treatment of osteoarthritis (OA) of the hand. Lasers Surg Med. 2005; 36 (3): 210-219.


  1. Hode L, Tunér J. Wrong parameters can give just any result. Laser Surg Med. 2006; 38: 343. (Letter to the editor).


  1. Bjordal J M. Can a Cochrane review in controversial areas be biased? A sensitivity analysis based on the protocol of a systematic Cochrane review Low Level Laser Therapy in Osteoarthritis. Photomed Laser Surg. 2005; 23 (5): 453-458.