Using Bad Statistics to Mislead

Dr. Ben Goldacre in Bad Science, pg. 186-187, wrote:

  • We can look at how these numbers  and calculations … are repeatedly misused and misunderstood.
  • Numbers … can ruin lives.


  • Newspapers like big numbers and eye-catching headlines (only newspapers? No, all those with vested interests love to indulge in them too! In fact this is the way they can mislead you and me!).
  • They need miracle cures and hidden scares… small percentage shifts … will never be enough for them to sell readers to advertisers.
  • To this end they pick the single most melodramatic and misleading way of describing any statistical increase …..

Example of How to Massage Data: Reporting of Relative Risk Reduction

Take a hypothetical case. Out of 100 men in their fifties with normal cholesterol, 4 will be expected to have a heart attack, whereas out of 100 men with high cholesterol, 6 will be expected to have a heart attack.

How to make a cholesterol scare?

If you put it in layman’s language (i.e. using the natural frequencies) there is no impact. Among those men with high cholesterol  only an extra of 2 heart attacks per hundred. No big deal, right?  Cholesterol will not scare you.

Here is how the professional of numbers or statisticians play their tricks on us.

It is equally right (mathematically speaking) to say that cholesterol increases the Relative Risk of heart attack by 50 percent!

This is how they massage the same data to make it more dramatic. Four men out of 100 will have heart attack with normal cholesterol, 6 men out of 100 if the cholesterol is high. The increase of 2 heart attacks out of 4.  You can then legitimately say cholesterol increase heart attack by 50 percent!

In chapter 2 of Honest Medicine Dr. Donald Murphy wrote:

  • Let’s consider the aspirin … a hypothetical study.  Researchers found that 10 of 1,000 volunteers who took one aspirin a day had a heart attack. They found that 20 of the 1,000 volunteers who took the placebo (sugar pill) had a heart attack.
  • How will the medical journals and the medial report this difference? How will the scientist and the media emphasize the importance of this finding? They will most likely report the relative risk reduction (RRR).
  • In this example, the RRR is an impressive 50 percent: going from 20 to 10 is a 50 percent change.
  • You get the impression that you can cut your risk of a heart attack in half by taking aspirin.
  • Take a closer look at these numbers. Only 10 out of the 1,000 volunteers taking aspirin benefit from this drug. The study also shows that 980 of the volunteers taking aspirin wouldn’t have a heart attack anyway because 980 of the volunteers taking the placebo didn’t have a heart attack. 10 of the volunteers would have a heart attack whether they took aspirin or not. The other 10 volunteers are the only ones who prevented a heart attack due to aspirin.
  • In this example, going from 2 percent (20 /1,000) to 1 percent (10/1,000) is only a 1 percent change. That is the likelihood that you would prevent a heart attack if you took aspirin — benefit of only 1 percent.
  • “Cut your risk of a heart attack by 1 percent” doesn’t have the punch of the headline as “ASPIRIN CUST RISK OF HEART ATTACK BY 50 PERCENT”.

Do you see how the benefit of 1 percent can be massaged and made to look great by “legitimately” turning into 50 percent relative risk reduction? Again let me emphasize, the stark reality is that out of the 100 people who take aspirin, only 2 people will benefit from it, in terms of preventing heart attack. Data presented in simple, raw form tells the truth more honestly!

A Word about Statistical Significance

When medical journals and the media report important findings, they refer to statistical significance. Statistics are based on probabilities, not on absolutes. A study that is statistically significant may not be clinically significant for you.

If a medical study reports that a finding is statistically significant, it means the finding is probably real and not just a matter of chance … science does not consist of only black and white facts. It is full of gray areas and can be very subjective …facts may not be so factual after all …. and health care is not black and white.

Things Get More Complicated

When prescribed medication or told to undergo chemotherapy, some patients are full of trust, taking for granted that this is best for them.  But some patients are more empowered. They wanted to know the possible outcome of the treatment. The want simple straight forward answers, as below:

For example when undergoing breast cancer treatment, you may wish to ask: For all the surgery, chemotherapy, radiotherapy (and tamoxifen) that you have been told to go for.

• How many patients were cured?

• How many died?

• How many survived after one, two, three, five or ten years after the treatments?

• How many contracted metastases of the liver, bone, lungs, etc.?

• Is there any correlation between the treatments they received and the metastases that occurred?

These questions and their answers are pertinent to all people. You need to know the answers to these basic questions, to be able to make some kind of informed decisions. Unfortunately when you need the medical literature, you will be carried away! Lost in medical or statistical jargons! You don’t get straight answers to the questions above.

Terminology Used to Clinical Trials

Oncologists use the term endpoint to refer to an outcome they are trying to measure with a clinical trial. Understanding endpoints is absolutely critical to understanding the technical medical literature. All journal articles reporting on clinical trials will report the results in terms of the endpoints which were measured. If you don’t understand what they mean, you can’t understand the article.

For example, oncologists frequently use the term “respond” to treatment; or they say, “you are responding to the treatment.”  Do you know what “respond” might mean?


Response is about measuring tumor shrinkage. Response is not used where the primary tumor has been removed surgically since in that case there are no detectable tumors to measure.

There are many kinds of responses:

  • Complete Response (CR): This means all detectable tumor has disappeared. A complete response does not necessarily mean the patient is cured. Even when no tumor can be seen on scans, there can be residual tumor which is too small to detect, and so unfortunately, complete responses may not last. A patient who has had a complete response may be said to be “NED”. NED means “No Evidence of Disease”.
  • Partial Response (PR): This roughly corresponds to at least a 50% decrease in the total tumor volume but with evidence of some residual disease still remaining. Partial responses aren’t usually cures and usually aren’t a long term benefit because significant tumor remains.
  • Minor Response (MR): This roughly means a small amount of shrinkage. Roughly speaking, a minor response is more than 25% of total tumor volume but less than the 50% that would make it a PR. A minor response is not enough to be considered a true response.
  • Stable Disease (SD): Stable disease means the tumors stay the same size or “insignificant” changes. This may include either a small amount of growth (typically less than 20 or 25%) or a small amount of shrinkage.  You may wish to know that some periods of stability are relatively common in some kinds of cancer even without treatment. Therefore, it is difficult to know if stable disease is the result of treatment. Claims of benefit for new treatments involving stable disease should be examined skeptically.
  • Progressive Disease (PD): Progressive disease means the tumor has grown significantly or that new tumors have appeared. The appearance of new tumors is always progressive disease. Progressive disease normally means the treatment has failed and in most cases is the signal that it’s time to try something else (or stop treatment altogether if no good options remain).
  • Objective Response (OR): Objective response means either a partial or complete response (In the literature you’ll frequently see “CR+PR” which means the same thing). When you see an objective response rate be sure to look at how many are complete responses and how many are partial since benefits from complete response tend to be greater. Often news reports and especially press releases by self-interested companies blur this and don’t reveal that the CR rate is low or non-existent. Track down the original source and find out!
  • “Clinical Benefit”: Clinical benefit is an informal term which usually means anything other than progressive disease. Use of this term is suspect, particularly if it is in a press release or news report. It isn’t automatically clear that patients with stable disease are benefiting from treatment since the natural history of cancer can include periods of apparent stable disease and since tumor shrinkage is not equal to clinical benefit to begin with.


Improvement in survival is generally considered to be the gold standard and is therefore a very important endpoint in cancer trials. It directly benefits patients.

Survival is an unambiguous end point that is not subject to investigator bias or interpretation. It is an end point that can be assessed easily, frequently, and without reliance on tumor measurements of any kind.

Therapies with a high treatment-related mortality might fail to show a survival benefit even if tumor control is substantially better with the new treatment.

Frequently, a big deal is made out of treatments which improve median survival by only a few weeks or months.

The common jargon used is OS, Overall Survival besides the different shades of survival.

Progression Free Survival

Progression Free Survival is the length of time you are both alive and free from any significant increase in your cancer (free from progression).

Disease Free Survival

Disease Free Survival is a special case of Progression Free Survival used as an endpoint in the clinical trials of adjuvant therapy to prevent recurrence after surgery to completely remove all visible cancer. In this case “progression” means the patient has had a recurrence.

Progression free survival and disease free survival can translate to an improvement in quality of life since symptoms from the cancer are delayed – but only if side effects of treatment aren’t worse.

Quality of Life

Quality of Life is supposed to measure how you feel and how you function. Although quality of life is certainly important in the broad sense, unfortunately, there is no unambiguous physical measurement or definable property which corresponds to your “Quality of Life”.

Quality of Life is therefore measured using a brief questionnaire in which patients rate their ability to function in various ways and enjoy life. Patients typically fill out the questionnaire several times during the course of the trial.

Quoted from: