Epidemiological terms are fast becoming part of everyday language. Terms like “flattening the curve”, “antibody tests” and “infection control” pepper conversations in supermarket queues and Zoom meetings.
Scientists must publicly share and explain, more now than ever before, the science behind efforts to stop the Covid-19 virus in its tracks. And, apart from some fake news and bad armchair epidemiology, it generally seems to work. In my experience at least, most people seem to talk about the virus with a decent scientific understanding of the basic mechanisms of transmission and the need to follow protective measures.
The communication of science to the public can however have negative consequences if the science underpinning the conversation is not as robust as it ideally should be. This risk is heightened in an emergency environment where not all the scientific method’s usual checks and balances are in place.
The rate at which scientific knowledge about Sars-CoV-2 (the virus that causes Covid-19) has been acquired is a remarkable collective achievement. Just five months ago no-one knew about the virus’s existence, but today it is the subject of study on an unprecedented scale.
In the face of a public health emergency, academic secrecy appears to have been suspended and scientific collaboration has increased. Leading medical journals have made their Covid-19 articles free-to-read for all. More than 200 clinical trials have been launched, in many cases uniting hospitals and laboratories from different countries. The hunt is on for a vaccine, a number of antiviral drug trials are on the go and new diagnostic tests are rapidly being developed and validated.
At the height of the HIV/AIDS epidemic in the 1990s, researchers and clinicians, spurred on by activists, rallied to combat the disease in what was then an unprecedented effort. But today’s technology and the pace of information-sharing surpasses what was possible 30 years ago.
Professor Tulio de Oliveira, a geneticist at KRISP in KwaZulu-Natal, said in a recent Data@breakfast webinar that “in just 100 days the scientific community has identified the virus, sequenced the genome, understood transmission, understood the proteins, and had an understanding of the receptor that enters the cell”. He said this came about because scientists decided to release their data quickly. “We talk about from sample to scientific results in days,” he said.
“This has completely changed the way we do traditional science.” de Oliveira said in a pre-Covid-19 world it could take up to 100 days from submission of a science paper to it being accepted for publication in a journal. “[Now]in that time we already have thousands of publications.”
Warning bells are ringing however about the poor quality of some scientific research in these pressurised times. The politicisation of science and hype in the media around clinically unproven drug treatments can create public confusion and fuel unrealistic expectations.
The closely related antimalarial drugs chloroquine and hydroxychloroquine recently became household names after US President Donald Trump touted them as proven treatments for Covid-19. This was after early trials suggested they may have some efficacy in the treatment of people with the disease. The public responded by buying up the drugs - in the process putting those who need them for rheumatoid arthritis and other auto immune conditions at risk.
Some scientists have warned against over-interpreting the early studies due to methodological flaws. Some hydroxychloroquine studies have since been abandoned, and some have suggested the drugs may increase the risk of cardiac arrest in patients with Covid-19 disease. While most scientists seem to consider hydroxychloroquine worth studying further, most also seem to agree that the jury is very much still out.
“The problem is when trial results come out and are presented to the media without important contextualisation, peer review and critique,” says Graeme Meintjes, an infectious diseases physician at the University of Cape Town. “A treatment (drug) needs to be safe and effective and focussed on patient outcomes. We are interested in saving lives, faster recovery and reduced hospitalisation.”
Too many small trials?
The urgency to do research and then to publish one’s findings as quickly as possible is entirely understandable given the severe toll Covid-19 is taking. But might the publication of findings from small, methodologically flawed studies end up doing more harm than good?
There is currently a deluge of small trials being registered which could result in low quality evidence, according to Helen Macdonald, research editor of the British Medical Journal(BMJ). In a recent “BMJ Evidence-based Medicine” podcast she said many of the trials could not be compared and was wasteful research.
“There is a need for less speed and more careful thought and for proper blinded randomised control trials done properly,” she said.
Andy Gray, a pharmacologist at the University of KwaZulu-Natal, has similar concerns. He says the new trend of fast-paced research could be risky in the future. “Speed will be valued over quality, with a weakening of the peer review process. That risks poor-quality research entering the ‘literature’ to a far greater extent than is already the case.”
Gray says “underpowered clinical trials” (meaning small trials with few participants) with inadequate processes for blinding and randomisation risk results that can be misleading or misinterpreted.
“When inclusion and exclusion criteria for participating in the trials are poorly determined, or applied, and clinician choice to alter allocations or regimens is allowed, the risks of biases are also increased.” This ultimately undermines the research study, he says. Gray adds: “When attempts are then made to synthesise the literature, we are left with only poor quality evidence on which to make decisions.”
Meintjes says there are an unprecedented number of similar hydroxychloroquine studies, addressing similar scientific hypothesis. “We now need larger studies,” he says, that will look at patient related outcomes.
As with HIV networks studies, Gray says, co-ordination and multi-centre studies for Covid-19 are possible, but these trials take time and have to be meticulously conducted and monitored.
Two large Covid-19 studies have been launched that should help avoid some of the pitfalls of doing many smaller trials with varying methodologies. These are both randomised controlled trials that will recruit large numbers of people to a standardised protocol that allows for a meaningful comparison of different Covid-19 treatments.
The United Kingdom’s RECOVERY trial has recruited more than 5 000 patients in over 165 hospitals and the SOLIDARITY trial, coordinated by the World Health Organisation, has four treatment arms and involves over 100 countries. Spotlight previously reported on South Africa’s participation in the SOLIDARITY trial and published an opinion piece on why such large, multi-arm clinical trials are needed for Covid-19.
One benefit of multi-arm trials such as RECOVERY and SOLIDARITY is that they allow us to compare treatments with each other and with placebo. Single-arm trials (where a treatment is not directly compared to any other treatment), on the other hand, do not allow for direct comparisons and accordingly provide less certain findings. They are however generally easier and quicker to do.
In a recent Covid-19 drug study reported in the New England Journal of Medicine (NEJM) for example, data were presented for "61 patients who received at least one dose of the antiviral drug remdesivir", but in seven patients there were no post-treatment data and one patient had a dosing error. There was also no control arm (in which one group of participants is not given the drug so they can be compared with the group receiving the drug).
Gray says although a University of Chicago Medicine study recruited more patients (125, of which 113 were described as “severe") in another trial of remdesivir, there was no control arm either, so no means of comparison.
The trial results, yet to be peer-reviewed and published, were leaked to the media after they were discussed at a university faculty meeting and soon after started making waves on Twitter.
The authors of an article published in the NEJM and which appeared after the remdesivir study, highlighted concerns about how evaluating and approving drugs can go awry during a public health crisis. They said: “It is a false dichotomy to suggest that we must choose between rapid deployment of treatments and adequate scientific scrutiny. For the Covid-19 pandemic and other pressing medical challenges, the health of individual patients and the public at large will be best served by remaining true to our time-tested approach to clinical trial evidence and drug evaluation, rather than cutting corners and resorting to appealing yet risky quick fixes.”
BCG: Correlation is not cause
Another study grabbing media attention in South Africa suggested a correlation between the century-old Bacillus Calmette–Guérin (BCG) vaccine and protection against Covid-19. The BCG vaccine is used in South Africa and various other countries with high TB burdens to prevent severe forms of TB in young children.
Following the publication of the study, Facebook and other social media platforms quickly ignited with BCG being declared a “game changer”.
But it is worth considering where the study was published. It was uploaded to the preprint server medRxiv, an online archive that shares academic research before it has been peer-reviewed and published in journals. In normal times, publication of such a study without peer-review would be frowned upon by many.
A number of critiques of the study have since been published (here is a particularly good one). One point some have made is that correlation is not cause and that there are other more plausible explanations for the correlation between BCG and Covid-19. It is also possible that the correlation will vanish as Covid-19 numbers increase in poorer countries with widespread BCG vaccination.
Given the compelling criticisms that have been published of the article, it seems unlikely that the article in its original form would have made it through peer-review at a reputable journal without very significant alterations. As such, the affair both raises questions over the need for scientific peer-review and over the media’s handling of studies that have not been peer-reviewed.