New study points to systemic shortfalls of scientific integrity in anti-doping

A new study by professor Roger Pielke Jr. from the University of Colorado, Boulder, USA, and professor Erik Boye from Oslo University Hospital, Norway, claims to document “a growing body of evidence which indicates that anti-doping regulation under WADA is sometimes arbitrary and too often not grounded in a solid foundation of evidence”.

In their research paper ‘Scientific Integrity and Anti-Doping Regulation’, which was published earlier this month by International Journal of Sport Policy and Politics, the two scientists focus on a key element of anti-doping policy implementation; the role of scientific evidence in anti-doping.

Based on a wide array of cases they conclude that WADA systematically falls short of basic standards of scientific integrity in anti-doping regulation:

“In the absence of reliable evidence, decision-making often becomes arbitrary, inconsistent and irreproducible, which threatens the integrity of anti-doping decision-making, the due process rights of athletes and the sustainability of anti-doping efforts,” they say.

“To the extent that we are correct in this assertion, it would be highly problematic because the entire structure of a successful anti-doping regime is based on securing scientific integrity.”

Their study focuses on shortfalls in standards of scientific integrity in four contexts:

· The prevalence of doping,

· Performance benefits and health risks,

· Errors and inconsistencies in accusation,

· The evaluation of anti-doping policies.

No reliable estimates of prevalence
The latest major research on the prevalence of doping estimated that about 50 percent of the athletes competing at the 2011 IAAF World Championship and 12th Pan Arab Games had doped.

But according to Pielke and Boye, neither WADA nor the International Association of Athletics Federations, IAAF, who commissioned the research, allowed the results to be published until six years later following substantial negotiations and a British parliamentary inquiry. Another recent study from 2015 estimated a prevalence of 14-39 percent.

“The prevalence of doping in elite sport is a key parameter in order to scale and formulate evidence-based anti-doping regulations. However, reliable estimates are not available and have been avoided and obstructed by anti-doping authorities,” Pielke and Boye say and argue that it would not necessarily be very costly to implement a rigorous programme to assess doping prevalence and how it changes over time:

“Whatever the total needed, it is miniscule in comparison to the now dated estimate of 130 billion US dollar in revenues from elite sports and 350 million US dollar devoted to drug testing.”

Prohibited List is arbitrary
WADA’s list of prohibited substances is regulated by rules given in the WADA Code and forms the basis of anti-doping regulation.

According to The Code, a substance or method shall be considered for inclusion on the list if WADA determines that the substance or method meets two of three criteria: If it has the potential to enhance or enhances sport performance, if it represents an actual or potential health risk to the athlete, or if it violates the ‘spirit of sport’.

“With respect to performance-enhancement or health risk, the language of The Code states explicitly that evidence is needed to meet the criteria. However, the third criterion, the ‘spirit of sport’, cannot be considered to be evidence-based,” the two scientists argue and conclude that “there is considerable evidence that the composition of the Prohibited List, which now totals more than 300 substances and methods, is neither rigorously evidence-based nor transparent”.

To illustrate their point, the two scientists give a number of examples in the paper, i.e.:

Tramadol is not on the Prohibited List, its performance-enhancing benefits are unknown, but it has clear health risks. Meldonium is banned, but its performance-enhancing benefits and its health risks are unknown. Letrozole is banned, but it has no performance-enhancing benefits and little health risks.

“At a minimum, these inconsistencies strongly suggest that decision-making with respect to including or not these drugs on the Prohibited List is arbitrary. Because WADA does not reveal the processes or evidence that it uses in making judgements about inclusion on the Prohibited List, it is impossible to know what, if any, evidence was used to support such decisions,” the two authors write.

False accusations
In the paper, Pielke and Boye present other alleged errors and inconsistencies in the current anti-doping regulation which they conclude have had profound impacts on many athletes’ careers.

One of the most severe examples involves the Saudi Arabian footballer Alaa Al-Kowaibki. In 2010 he was suspended one year for doping. But the WADA laboratory in Malaysia that had performed the analyses was subsequently suspended by WADA for producing numerous false positive test results.

The Malaysian lab protested the suspension and appealed the decision to CAS. WADA won the case based on evidence in the cases of six athletes wrongly accused of doping based on flawed analyses conducted by the lab. One of the six athletes was Alaa Al-Kowaibki. But neither WADA nor the Saudi Anti-Doping Organization told the footballer that his sample was considered a false positive and that he had wrongly served a doping suspension.

“Remarkably, it was only in 2017 that Al-Kowaibki learned that he was in fact cleared years ago by CAS. This case shows that the anti-doping system requires more than just solid evidence, it also requires that such evidence be used to secure the due process rights of athletes who are occasionally falsely accused,” Pielke and Boye argue.

A profound bias
Pielke and Boye also point to another example of what they argue is an unfair differential treatment in WADA’s International Standards for Laboratories.

While negative doping samples can be stored and re-tested years after they were taken if a new testing method is developed, positive drug tests only need to be stored for three months or until the completion of any immediate challenge or investigation of the testing result.

For this reason, athlete claims of innocence cannot be validated with a re-examination of the original sample if new information or testing procedures are developed later-on.

“This situation creates a profound bias against athletes and the use of science to resolve anti-doping disputes,” Pielke and Boye say, calling for symmetrical demands for the storage of doping samples.

Guilty until proven innocent
As in most democratic systems of jurisprudence, athletes are presumed to be innocent of doping until proven guilty. But this presumption reverses to guilty until proven innocent when an athlete tests positive, the two scientists point out.

WADA’s International Standards for Laboratories presume that the labs have conducted sample analyses and custodial procedures in accordance with the standards and explain what this means: The burden is on the Athlete or other person to establish, by a balance of probability, a departure from the International Standards for Laboratories that could reasonably have caused the Adverse Analytical Finding.

“If anti-doping administration cannot afford to maintain the same presumption of innocence for athletes typically found in common law, then anti-doping regulation is to some extent sacrificing the rights of some false accused athletes in order to facilitate the sanctioning of those who test positive, whether innocent or not,” Pielke and Boye argue.

Lack of evidence and research
Furthermore, the two scientists claim that policy interventions preferred by anti-doping organisations to fight doping among athletes do not rest on a firm foundation of evidence. Despite the lack of evidence, education is often highlighted as a key approach to anti-doping.

“In 2007, WADA supported a review of evidence of the efficacy of educational strategies in response to doping in sport. That review does not support the contention that anti-doping education is an effective strategy of motivating anti-doping behaviors,” Pielke and Boye state and note that a follow-up assessment commissioned by WADA in 2015 also concluded that little is actually known about what works.

“Without reliable information on doping prevalence and how that metric changes over time, it will be very difficult, if not impossible, to arrive at a robust understanding of what works in anti-doping policy.”

Actions are needed
Pielke and Boye recommend the formation of a formally recognised group of independent scientists monitoring the prevalence of doping with its results and methods routinely and rapidly published in the public domain:

“Until anti-doping organizations take prevalence monitoring seriously, it will be difficult to take anti-doping organizations seriously.”

The two scientists also recommend a reduction of the size of WADA’s prohibited list and to strengthen due process by equal treatment of positive and negative samples, maintenance of the presumption of innocence throughout the arbitration and sanctioning process, a balancing of the weight of evidence presented by the WADA-accredited laboratories with independent expertise, and the creation of mechanisms to correct flawed sanctions when evidence indicates that mistakes were made:

“There are structural and procedural biases against athlete rights found in WADA procedures for sample storage, presumption of innocence and burden of proof.”

They also find that there should be more attention to the consequences when organizations violate the anti-doping rules, and that actions are needed to conduct a more rigorous policy evaluation:

“If anti-doping is in fact to work then it should be able to not just quantify prevalence of doping, but to quantify the effects of various interventions designed to manage or even reduce doping in sport.”

Not just science
Play the Game has asked WADA to comment on Pielke and Boye’s critique.

“Scientific rigor and robustness are of primary importance to and a constant priority for WADA. As a matter of course, WADA consults external experts to consolidate the scientific basis of cases prosecuted by the Agency in front of the Court of Arbitration for Sport, CAS,” says Maggie Durand, WADA’s manager of Media Relations and Communications, in an email.

“Beyond science, it is important that the unique contexts and legal dimensions of each case being managed are taken into account. It should be noted that WADA is not always a party to cases and therefore reviews such cases for consistency with the rules of the Code and not necessarily from a scientific perspective.”

Maggie Durand also points out that the scientific elements underlying the anti-doping system, including the inclusion of substances on the Prohibited List, the establishment of decision limits, analytical methods, etc., are devised and revised by commissions of independent world-leading scientists.

“Indeed, decision limits and analytical methods are routinely subject to peer review and, pursuant to the World Anti-Doping Code, only benefit from presumption of scientific validity if that is the case.”

According to Durand, challenges to the scientific elements underlying the Code are frequently made but rarely succeed:

“WADA has in recent years successfully defended many scientific challenges before CAS. On the very rare occasions where CAS has upheld any aspect of a scientific challenge, WADA has reviewed the situation and undertaken any necessary action to provide the scientific information required and, if necessary, adjusted the regulation, International Standards or technical Documents, in order to reflect the decision.”

Scientists: Disappointing
Even though Maggie Durand also notes that WADA since 2001 has committed more than 80 million USD “to help researchers around the world develop breakthroughs in anti-doping science”, the two scientists behind the new research paper are disappointed with WADA’s reply:

“They do not address our items of critique, but present a general statement about how good they are,” says Erik Boye.

“The issues that we raise are legitimate and supported by evidence. It is disappointing that WADA refuses to acknowledge these issues, but WADA’s failure to engage them helps to confirm our concerns about scientific integrity in anti-doping,” says Roger Pielke Jr..