Practicing hospital physician here. We never use the term "accuracy" to describe diagnostic tests, and whenever I see that term used, I'm very suspicious of the claim being made. In other words, this Bosch announcement can probably be ignored.
We use the terms "sensitivity" and "specificity" to describe diagnostic test performance, and never to my knowledge talk about a test's "accuracy." Usually accuracy refers to the sensitivity, but there's a lot of liberty taken with that term in marketing talk.
For example, the existing PCR test being used does not have ideal sensitivity, which for practical purposes means that there is a substantial false-negative rate. Which is why (among other reasons) we weren't testing asymptomatic patients in the early phase.
According to Wikipedia, accuracy is actually defined in this context. This does not not mean that Bosch has not used everyday speech here but it is not clear that they have. Too bad that the article does not include links to additional information.
Of course accuracy is defined, but it's not the right metric because you usually have a large bias in the tested population: even if you only test people with symptoms, you might only have 10% of people that actually have the disease; so a test that always says "NO" would have 90% accuracy.
That's why you look at True Positive Rate (sensitivity) and True Negative Rate (specificity) to factor that bias out.
If TPR is 95%, that means that 95% of the people that are actually infected (10% of the pop in the previous ex.) will be detected as such.
IF TNR is 20%, that means you'll mistakenly detect 20% of people that don't have it as infected, which would be really bad.
In other fields, people use Precision and Recall. Precision = % of times the test is right when it predicts a positive. Recall = how many of the real positives were detected as such.
> IF TNR is 20%, that means you'll mistakenly detect 20% of people that don't have it as infected, which would be really bad.
Would it though? It's my understanding that most people who test positive are still being sent home, not forcefully hospitalized. And given that we don't actually have a widely used pharmaceutical intervention protocol, there's no side effects to worry about from false positives (eg of the opposite of this: if we were giving every confirmed case chloroquine or interleukin). The worst case outcome from a high false positive rate (low TNR) would seem to me that people would be extra good at self quarantining.
If you're a healthcare worker, the cost of a false-positive could be high.
For them and others one cost is that they'll feel themselves somewhat immune when they're not and may not take precautions. It will also confuse study of the disease if it starts to look like your "2nd" infection can be much worse than your first.
Hospitals in an epidemic are major spreaders of the infection. If false positive results lead to patients being put together with real carriers, they will soon be infected.
From a recent report[1] from Italy:
> For example, we are learning that hospitals might be the main Covid-19 carriers, as they are rapidly populated by infected patients, facilitating transmission to uninfected patients. Patients are transported by our regional system, which also contributes to spreading the disease as its ambulances and personnel rapidly become vectors. Health workers are asymptomatic carriers or sick without surveillance; some might die, including young people, which increases the stress of those on the front line.
It could mean that some of the major causes of the large death toll in Italy are the lack of personal protection equipment, and, possibly, laxity in following protocol.
> some of the major causes of the large death toll...
Wrong. The major cause is that the number of people getting sick at once was much higher than the infrastructure can handle. That's what other governments saw and now try to avoid with the lockdowns.
This was made 10 days ago by the Italian biotech Association it's the number of new cases and deaths per week (red), compared with the number of cases per week at the peak of the "normal" flu (blue):
It can happen everywhere if the growth continues, simply because nothing can cope with fast exponential growth. At the moment, wherever it is uncontrolled, it's around 3 days to double. Sounds like small numbers, just 2 and 3? That gives however a thousandfold growth in 30 days: everybody working with computers should be very familiar with the equation 2^10 = 1024 (it's 2^(30/3) == 2^10 == 1024, 30 being the number of days for the projection, 3 the doubling time and 2 the doubling itself).
Nobody has thousandfold more hospital beds and doctors ready, even less a million times more, which is the two month's growth.
Discussing other factors without first admitting the major one is obviously biased.
> there just aren't enough people to sustain two months' growth
It can be simply calculated: it is expected that without measures the growth to 70% of population would be continuous (very approximately, the end phase wouldn't, we're estimating the limit). So the target is 6e9 people. If we assume that 4/5ths are the people who remain undetected by our current sampling, we want to know the growth between the current known infected and the target which is then 1.26e9 people. Currently known are 0.5 million infected. So the fastest end of growth phase would be just: 2520 times or around 2^11=2048 == just 11 times 3 day doubling time, or 33 days.
The growth will surely not be always exactly 3 days however, so it will be slower, but still not less dramatic, because the resources are many, many times smaller, in the poorer countries many tens of times smaller.
In short it can be very, very bad, and that will be much longer than just a month, just not the exact growth as now.
See the papers from Imperial College London for the exact shapes of the curves and the examples of their speed and growth.
That may be highly specific to this period, where italy outside hospitals is in lockdown. As we move forward, people will use the tests to decide whether to go back to work, which makes them more critical
The substantial false-negative rate is also one of the reasons why covid patients weren't released from quarantine after a first negative test, since a single result doesn't pass the risk threshold. Multiple negatives in sequence give the desired certainty of someone really being covid-free.
Thank you! I've been bringing up the sensitivity issue a lot recently, since it seems to be disregarded by many upset at the situation. I have a question for you if you don't mind: when so many people complain about lack of testing in the US, do you know what false-negative rates would exist for the tests we'd expect to see (like what they see available en mass in other countries)? And would you know what that level would need to be for the test to be useful?
No, accuracy is the same here it is for any binary classification. accuracy = (true positives + true negatives) / all samples. A largely useless measure. As a single score F1 score is much better, but sensitivity and specifity are specified separately for good reason. It makes napking math much easier, for one thing.
What confused me is there are actually two definitions of accuracy.
Accuracy: ACC = (TP+TN)/(P+N)
Balanced Accuracy: BA = (TPR+TNR)/2
The definition I gave is the second not the first. You are probably correct that the conference abstract I linked to is using it in the first sense not the second. (I'm not 100% sure though.)
So would sensitivity be the percentage of infected people who test positive and specificity be the percentage of non-infected who test negative (i.e. sensitivity = (1 - false negative probability) and specificity = (1 - false positive probability))?
Yes. The intuitive way I try to remember it is to think that, in just plain English, "How sensitive is this thing?" means "How likely is it to detect whatever it's supposed to detect?" (e.g.: How likely is this security camera to detect movement?)
"Sensitivity" is an inaccurate term, since for non-native English speakers, it could be misinterpreted as a term to mean some kind of emotional response.
"Accuracy" is a bit more scientific- leaning and therefore easier to translate.
Remember, COVID-19 is a global phenomenon, its not just relegated to the Anglo-sphere. Bosch, being a German company, is probably a bit sensitive to the accuracy of translations that will be needed to market this thing effectively around the world .. ;)
> Remember, COVID-19 is a global phenomenon, its not just relegated to the Anglo-sphere. Bosch, being a German company, is probably a bit sensitive to the accuracy of translations that will be needed to market this thing effectively around the world .. ;)
...what?
I was only trying to help the parent find a way to remember which one is sensitivity and which one is specificity...
Yes, and you perfectly demonstrated the confusion possible by selecting the wrong terms. Sure, 'specificity' and 'sensitivity' are scientific terms - but for marketing material, can be confused with emotional responses. Thus, I believe Bosch chose 'accuracy', since this translates into other languages more effectively.
...marketing material? To the non-English world? I was literally just explaining a potentially helpful thought process to an English-speaking HN visitor dev who lists his(?) public key on his profile... not trying to produce marketing material for a French waiter or something. Are you sure you're not the one confused here rather than him?
I'm only trying to point out that Bosch may have specifically chosen the term 'accuracy' instead of 'specificity' or 'sensitivity', which don't translate easily outside of the Anglo-sphere, and in my opinion your thread is a valid example of why those two terms don't get used to market this device, which is what kicked this thread off in the first place ..
It is more likely that they were trying do deliberatly write a vague statement, their (native) German announcement uses the same "Genauigkeit", instead of the relevant terms "Sensitivität" and "Spezifizität".
Bosch sells its releases in 100's of different countries. I'm not saying they weren't using inaccurate terminology intentionally, more that there are indeed some guidelines for how things are couched in releases of this nature and one of those guidelines is translatability.
Especially apropos life-support systems where, indeed, RTFM||die is a thing.
I don't really know data science but in assay validation, sensitivity basically measures false negative rate, and specificity measures false positive rate. Accuracy and precision are measures of assay consistency or tightness. (statisticians may not purely agree with my simple explanation)
These terms don't translate very well on an International market. Bosch probably chose its terminology to suit its plans to market these devices outside the Anglo-sphere, as well...
We use the terms "sensitivity" and "specificity" to describe diagnostic test performance, and never to my knowledge talk about a test's "accuracy." Usually accuracy refers to the sensitivity, but there's a lot of liberty taken with that term in marketing talk.
For example, the existing PCR test being used does not have ideal sensitivity, which for practical purposes means that there is a substantial false-negative rate. Which is why (among other reasons) we weren't testing asymptomatic patients in the early phase.