PDF version of this entire document

Sensitivity

It is useful to compare the performance of different measures of model quality. For a given measure, the level of model degradation that can be detected in the validation experiments described above depends on both the change in the value of the measure as a function of model degradation and the uncertainty in the value. To quantify this, we define the sensitivity $D$ of a measure as follows.

$$D(m,d) = \frac{1}{\bar{\sigma}}\left(\frac{m(d)-m(0)}{d}\right),$$ (7)

where $m(d)$ is the value of the measure for some degree of degradation $d$, $\overline{\sigma}$ is the mean error in the estimate of $m$ over the range. $D(m,d)$ is reciprocal of the change in $d$ required for $m(d)$ to change by one noise standard error, which indicates the lower limit of change in quality $d$ which can be detected by the measure. Sensitivities for the specificity and generalisation for different values of shuffle radius are shown in Figure 8. These results demonstrate that specificity is a more sensitive measure of model quality than generalisation, and that the use of shuffle distance improves the sensitivities of both measures over those obtained using Euclidean distance.

Figure 8: The sensitivity of Specificity and Generalisation