Search
Search type

Methods@Manchester

Risk Scores in the Social Sciences

Social scientists are interested in associations between explanatory variables measured at an earlier point in time and later outcomes. An estimated model that relates a binary outcome to a set of predictors is known as a risk score. The main motivations for constructing risk scores are to classify and to predict and thus to aid decision making in conditions of uncertainty.

Construction and use

The thinking that lies behind the construction and use of risk scores might be applied more widely. A common approach, often found in developmental psychology for example, is to relate a set of childhood experiences and circumstances to psychopathological adolescent and adult outcomes.

Variables

Researchers establish that some variables, conditional on the effects of others, are statistically significantly related to the outcome.

This approach can be useful but, by concentrating on the estimates for particular predictors and some measure of model fit it is, at best, incomplete because we do not learn how well we can predict both the good and the poor outcomes, nor how we might use the risk score to target (or to discourage) future interventions.

Summarising information

A popular way of summarising the information about classification is to estimate a receiver operating characteristic (ROC) curve as shown in the below Figure. It is then possible to generate summary measures from the ROC curve that can be used to compare the predictive capacity of different models.

Importantly, the framework that generates the ROC curve can also be combined with information about the costs and benefits of different decisions to help to determine how best to target interventions.

Graph
Fig. : ROC curve with diagonal line indicating no discrimination between poor and good outcomes

Key publications

  • Krzanowski, W. J. and Hand, D. J. (2009). ROC curves for continuous data. Boca Raton, Fl.: Chapman and Hall/CRC.
  • Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford: OUP.
  • Swets, J. A., Dawes, R. M. and Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest, 1, 1-26.

The methods are being applied in analyses of attrition in longitudinal studies in a project funded by ESRC as part of its Survey Design and Measurement Initiative.

PDF slides

Download PDF slides of the presentation 'What are Risk Scores?'