top of page
Search
Writer's pictureJ. Falatko D.O.

Random Tests and A Cognitive Cartwheel

John Falatko D.O.


I was on X (twitter) the other night getting small dopamine hits when and came across a post taking a shot at the intelligence of physicians. It was clearly an engagement trap, but I couldn’t help but get flustered. The author of the post wrote, “95% of medical students get this wrong. Doctors don’t know anything and are mainly midtwits walking around telling everyone what to do.”

 

The question proposed in the post is a common medical riddle.  It goes something like this:

 

            “If the prevalence of a disease is 1/1000, and a test to detect this disease has a 5% false positive rate, what is the likelihood that a positive test is a true positive?”

 

Most students/doctors would answer 95% and move on with their lives. An astute test taker would say to themselves, “This is too easy, I must be missing something.” Then commence in mental gymnastics. 

 

The question is written to trick the reader. It lacks the expected logical progression that our minds use to reason a complex topic. It is in the spirit, for example, of the convenient store riddle:

 

“A baseball and baseball glove are purchased at the convenience store.  The baseball costs $1 less than the baseball glove. The total purchase was $1.10.  How much was the baseball?

 

Most people would answer 10 cents. Which is incorrect. The baseball cost 5c and the baseball glove $1.05. But you don’t care, because the answer is trivial and irrelevant to your survival, so the answer that comes the quickest is the one that’s regurgitated. No need to call anyone a midtwit.

 

Like the baseball and glove the answer to the medical question goes as follows.  Of the 1000 sampled, there will 51 positive tests.  50 positive tests for the error rate of 5%, and one test for the actual person that has the disease. In a random sample of 1000, the likelihood is roughly 2% that it is a true positive (1/51).

 

Like the baseball and glove question, the medical question is trivial. It is a test of reasoning, not of medical acumen. In fact, you don’t even need a science background to answer it. An understanding of probability and basic mathematics is what is all you need. So, most doctors/medical students put down the first thing that comes them and move on.

 

Written in such a way that follows logical steps would allow one to consistently answer correctly. A basic logical progression goes something like this: 1) This occured, 2) Then this, 3) Resulted in this, etc.  More plainly: If X, then Y, if Y then Z, therefore if X then Z. Ta Da!.

 

Allow me to rephrase:

 

If are 51 positive tests in a random sample of 1000 patients. The test has an expected false positive rate of 5% (0.05 x 1000 = 50 false positive results).  There is only 1 patient of the 1000 that has the disease.  What are the odds that one of the positive tests represents the patient with the disease?

 

Very few medical students would get this wrong.  I would bet that very few high school sophomores would get this wrong.

 

Even if we were to address cognitive challenges by leaving out important sequential steps, the original question would have no application to daily practice. Not only is it cognitively deviant; it is not practical.

 

The reason why it has no practicality is because we do not order tests at random. The first step is to gather information, like symptoms and a physical exam. As new information becomes available, the probability, or odds, that the disease is present changes. Let’s use a real-life example like Lupus.

 

Lupus would work well in the example because it has a prevalence of somewhere between 1/1000 to 1/10,000 women. Lupus also has an imperfect screening test known as an ANA (anti-nuclear antibody).  ANA’s have a false positive rate in the general population of roughly 5-10% (for simplicity we'll call it 5%). So, if I were to randomly screen 1000 women with ANAs, it is likely I would get about 50 positive results, with one woman having the disease.

 

However, if you were in my office for anything other than a routine physical, you would likely be having symptoms.  With each symptom consistent with the disease, the odds that you have it increase. If you are complaining of rash across your chest and face, migratory arthritis with intermittent synovitis, and brown colored urine, your odds go way up. The classic signs of Lupus are a malar rash, intermittent painful swollen joints, and progressive glomerulonephritis (renal failure).  If you have all three, the probability you have the disease is high, let’s call it 80%.

 

Going back to our example:  If I test 1000 patients with the above constellation of symptoms with the same test (5% error rate), there will be 850 total positives. Of those, 800 will be true positives and 50 will be false positives. Calculate that out and you get 94% probability of a a true positive result.  The test performed as advertised.

 

See…your doctor is not an idiot. If they’ve received a decent education or been taught by yours truly.

 

Now, if you show up to your doctor’s office requesting a battery of tests because of some non-specific symptoms, then you may fall into the first scenario. There’s a good chance something will be positive, it will likely be a false positive, and you will be headed down the trail of uncertainty, anxiety, and prescriptions.

 

This happens in the real life, Lupus and rheumatoid arthritis are known to be inappropriately diagnosed diseases because of this phenomenon. But hey, Enbrel may make your joints feel a little better, and it’s a heck of a placebo.  Don’t worry about the increased risk of lymphoma, or tuberculosis.

 

You may be wondering, “Why would my doctor order the battery of tests I requested even if they know they will not perform well?”

 

That’s a complex question to answer…In an attempt to distill it down; it’s easier to order the tests and deal with the chance of a false positives, then to explain the cognitive cartwheel above in a 20-minute office visit. Believe me, I’ve tried. When I first started, I wasn’t that busy. I had more time explain things like this. Even when there’s no benefit, and could be harm, I would be petitioned to order the tests.  Sometimes you just need to see for yourself.  I’m not immune to this. The default when dealing with uncertainty is to gather more information, even if the information may be unreliable.

 

As a patient that is not feeling well, that may get a battery of tests, you can do yourself a favor by scheduling a follow-up appointment to discuss the results.  Please, don’t try to short-cut the interpretation by sending a Mychart message or asking the nurse to explain things to you over the phone.  You can also save yourself a lot of trouble by asking your doctor one simple question, “How much money would you put on me having “x” disease?”  If your doctor says, $0, or spends the next couple minutes with some wishy-washy response, they don’t believe the test, and you shouldn’t either.

13 views0 comments

Recent Posts

See All

You Really Should Go to Church

Last week I was in New York City. My first time. It is a massive place. Far larger than I imagined. I’ve been to “large” cities before....

Comments


bottom of page