Follow-up data

Predicting remission

The most straightforward target class is that of remission. This is a binary attribute stating whether the disorder is still present during the second test, or whether the disorder has gone into remission (ie., the test indicates the patient is no longer suffering from depression). We will be using J48, an algorithm to generate decision trees based on selecting the most informative attributes. Other algorithms have been considered, such as JRip (rule-based), NNge (nearest-neighborlike rule-based) and NBtree (tree with naive bayes classifiers at leaves) but they did not perform as well or their results were harder to interpret.

The following attributes have been removed so as not to confound the results:

ID, MADRS.2, BDI_[...].2, SOURCE, Rverschil, remissie_MADRS, quintiles_MADRS, quintiles_BDI.

One problem is that there is an uneven distribution of patients having remission and those that do not (the latter represents 80% of the instances). Regardless of the actual remission rates with depression, it seems that the class of patients without remission is over-represented. The results of this is decision trees like this:

J48 pruned tree
------------------
: geen remissie (601.0/115.0)

Such a tree will, of course, perform as well as the prior probability of "no remission." Note that the preceeding decision tree represents a pruned tree. The unpruned version contains many more branches and actually performs better on predicting remission, although it has a lower overall performance -- which is what pruning optimizes over. This means that pruning could run counter to our purposes. The confusion matrix for an unpruned tree on the data as-is (classifying for remission_BDI):

a b <-- classified as F-Measure
414 72 | a = geen remissie 0.836
91 24 | b = remissie 0.227

This outcome is, prima facie, not very hopeful. From the instances which are predicted to undergo remission, the majority will actually be an erroneous prediction! Other algorithms, such as rule-based learners, display similar performance. This means that it is not meaningful to compare different methods using the overall performance unless it exceeds the prior probability of no remission; we will employ confusion matrices instead.

There are two ways of dealing with this problem. The first is to resample the data with a uniform class distribution, the other is to specify a higher misclassification cost for instances incorrectly classified has not having remission. Both appear to be problematic. Resampling could be problematic because that data being trained on has an unrealistic distribution. Cost-sensitive evaluation might be problematic because the costs might be rather arbitrarily chosen to fit the sample.

Results using FilteredClassifier with spread subsampling with a uniform, 50-50 distribution of remission and no remission (note that the filter was only applied to the training data, hence the unchanged number of instances in the confusion matrix):

a b <-- classified as F-Measure
317 169 | a = geen remissie 0.746
47 68 | b = remissie 0.386

(Percent correct equals 64%). This result is much better than the previous unpruned tree, because the number of correctly predicted remissions is more than twice as large. However, the problem of incorrectly predicted remissions has increased!

Similar tricks can be performed using cost-sensitive classification. Here we choose the MetaCost classifier with a cost matrix which punishes classifications for the over-represented class, hoping to obtain a more balanced result:

Cost Matrix a b <-- classified as F-Measure
0 1 430 56 | a = geen remissie 0.859
3 0 85 30 | b = remissie 0.229

(Percent correct equals 76%). It is apparent that a trade-off can be made between false positives and false negatives. Unfortunately it seems that reducing one increases the other. Endless fiddling with cost matrices probably increases the risk of overfitting to the data at hand.

The full decision tree:

BDI_TOT.1 <= 24

| jaarmet1 <= 2.005

| | ernstdephe = onbekend

| | | MADRS.1 <= 18: remissie (3.0)

| | | MADRS.1 > 18: geen remissie (5.02)

| | ernstdephe = gedeeltelijk in remissie: remissie (1.26/0.26)

| | ernstdephe = licht

| | | midafve <= 0

| | | | hoofdiagnose = somatoforme stoornis: geen remissie (1.26/0.26)

| | | | hoofdiagnose = status afwezig: remissie (3.0/1.0)

| | | | hoofdiagnose = stemmingsstoornis: remissie (24.0/5.0)

| | | | hoofdiagnose = angststoornis: geen remissie (6.51)

| | | | hoofdiagnose = geen klinische diagnose: geen remissie (1.0)

| | | | hoofdiagnose = anders: remissie (2.0)

| | | midafve > 0: geen remissie (2.0)

| | ernstdephe = matig

| | | diagangst <= 0

| | | | BDI_TOT.1 <= 17: remissie (9.41/2.37)

| | | | BDI_TOT.1 > 17

| | | | | specfob <= 0

| | | | | | kindthuis = nee: geen remissie (23.08/4.21)

| | | | | | kindthuis = ja

| | | | | | | BDI_SOM.1 <= 9: remissie (3.05/0.05)

| | | | | | | BDI_SOM.1 > 9

| | | | | | | | BDI_AFF.1 <= 4: remissie (3.43/0.43)

| | | | | | | | BDI_AFF.1 > 4: geen remissie (7.49/1.0)

| | | | | specfob > 0: geen remissie (5.84)

| | | diagangst > 0

| | | | BDI_AFF.1 <= 1: geen remissie (2.09)

| | | | BDI_AFF.1 > 1

| | | | | BDI_AFF.1 <= 4: remissie (13.55/1.55)

| | | | | BDI_AFF.1 > 4: geen remissie (3.13/1.0)

| | ernstdephe = ernstig zonder psychotische kenmerken: remissie (6.26/0.26)

| jaarmet1 > 2.005

| | changemod1 = additie

| | | VOLG.1 <= 2.88: remissie (2.0)

| | | VOLG.1 > 2.88: geen remissie (3.0)

| | changemod1 = geen wijziging bekend

| | | GEMPH.1 <= 2: remissie (22.47/0.47)

| | | GEMPH.1 > 2

| | | | tijdROM12wek <= 16.43: remissie (2.04/0.04)

| | | | tijdROM12wek > 16.43: geen remissie (2.0)

| | changemod1 = switch: geen remissie (1.0)

| | changemod1 = additie + switch: remissie (1.0)

BDI_TOT.1 > 24

| BDI_AFF.1 <= 5

| | BDI_COG.1 <= 9: geen remissie (44.98/0.47)

| | BDI_COG.1 > 9

| | | specfob <= 0

| | | | AFFLAB.1 <= 3.75

| | | | | ID <= 10004060: geen remissie (5.78/0.4)

| | | | | ID > 10004060: remissie (7.15/0.13)

| | | | AFFLAB.1 > 3.75: geen remissie (19.05/0.98)

| | | specfob > 0: geen remissie (7.76)

| BDI_AFF.1 > 5: geen remissie (381.4/0.61)

Another problem is that there are two possible outcomes to predict: remission according to MADRS, and remission according to BDI. A possible solution is to filter the data to retain only those instances where the remission according to MADRS and BDI is equal, ie., to filter out conflicting outcomes. This removes about 17% of the original instances. It did not, however, improve the results significantly at first glance.

A last straw was to employ boosting. Boosting attempts to improve the performance of a weak algorithm by repeatedly training on misclassified instances. This theoretically allows algorithms to approach perfect performance, given enough iterations. In this case however it did not yield significantly better results, and has the added downside of producing numerous cascading decision trees which are more difficult to interpret than a single decision tree. The results, using AdaBoostM1 on a uniform spread subsample (75% percent correct):

a b <-- classified as F-measure
315 171 | a = geen remissie 0.744
46 69 | b = remissie 0.389

Predicting improvement

Instead of predicting which patients will go into remission, it is also interesting to predict which patients generally improve. The difference is that this will include patients going from severe to mild depression. A way to do this is to group the last four remission quintiles into one class (these show improvement), and contrast it with the group in the first quintile (who show little to no improvement according to the tests). Although patients generally improve over time, whether treated or not (Posternak et al, 2001; as cited on Wikipedia), it can be useful to discover what kind of patients form an exception to this tendency.

Results using uniform spread subsampling (65% percent correct):

a b <-- classified as F-measure
348 122 | a = twee_vijf_drie_vier 0.777
78 39 | b = een 0.281

The decision tree:

paniekstmag <= 0

| midafve <= 0

| | switch1far = niet van toepassing

| | | BDI_COG.1 <= 2: een (8.84/0.92)

| | | BDI_COG.1 > 2: twee_vijf_drie_vier (156.21/53.38)

| | switch1far = snri: een (3.49/0.31)

| | switch1far = geen

| | | jaarmet1 <= 2.004: twee_vijf_drie_vier (2.55/0.19)

| | | jaarmet1 > 2.004: een (4.43/1.27)

| | switch1far = tca: twee_vijf_drie_vier (0.0)

| | switch1far = snri en benzo: twee_vijf_drie_vier (0.0)

| | switch1far = ssri: twee_vijf_drie_vier (2.32/0.12)

| | switch1far = tca en benzo: twee_vijf_drie_vier (1.16/0.06)
| | switch1far = benzodiazepinen: twee_vijf_drie_vier (0.0)

| midafve > 0: een (9.0/2.0)

paniekstmag > 0

| jaarmet1 <= 2.003: twee_vijf_drie_vier (2.0)

| jaarmet1 > 2.003

| | BDI_AFF.1 <= 3: twee_vijf_drie_vier (3.0/1.0)

| | BDI_AFF.1 > 3: een (9.0)

After some attempts it appears that this classification task is even more difficult than the previous one.

Attribute selection

Besides class prediction it is also interesting to look at what subset of attributes is most predictive of the actual class. The simplest method of doing this is to iteratively add the attribute most correlated with the class to evaluate on, while simultaneously selecting attributes which are least correlated with those already selected. The result is a small subset of relatively independent and predictive attributes:

Attribute Subset Evaluator (supervised, Class (nominal): 93 remissie_BDI):

CFS Subset Evaluator

Selected attributes: 2,3,4,5,23,34,46,47,54 : 9

MADRS.1
BDI_AFF.1
BDI_COG.1
BDI_SOM.1
wijz1psy
switch2psy
COGVER.1
IDENT.1
NINTIM.1

The test results (MADRS, BDI) are indeed good predictors of remission, but the rest of the attributes are a somewhat arbitrary selection (the peformance of this set of attributes was not noteworthy). Much of the other attributes might have been equally important predictors, such as attributes describing addictions, other disorders and marital and job status. The fact that these are not part of this attribute subset demonstrates the difficulty of making non-arbitrary inferences from this data.

This prior knowledge of which attributes are relevant also allows us to make a manual selection of attributes, and see how it performs:

Selected attributes:

MADRS.1
BDI_AFF.1
BDI_COG.1
BDI_SOM.1
burgstaat
kindthuis
opleid
werk
aantdiag
ernstdephe
aantangst
alcafhe
deprrecid

Using J48 with uniform spread subsampling, the performance is 67.7%:

a b <-- classified as F-measure
344 142 | a = geen remissie 0.78
52 63 | b = remissie 0.394

J48 pruned tree

------------------

BDI_AFF.1 <= 5
| BDI_SOM.1 <= 12
| | deprrecid = enkelvoudig: remissie (20.69/2.81)
| | deprrecid = recidiverend
| | | BDI_COG.1 <= 11: remissie (32.74/7.0)
| | | BDI_COG.1 > 11: geen remissie (3.07/0.07)
| BDI_SOM.1 > 12
| | deprrecid = enkelvoudig
| | | MADRS.1 <= 25: geen remissie (4.51/1.0)
| | | MADRS.1 > 25: remissie (4.56)
| | deprrecid = recidiverend: geen remissie (26.51/8.51)
BDI_AFF.1 > 5
| werk = ziektewet/WAO: geen remissie (46.03/7.56)
| werk = geen betaald werk
| | aantangst <= 0: geen remissie (22.27/5.19)
| | aantangst > 0
| | | deprrecid = enkelvoudig: geen remissie (6.0/2.0)
| | | deprrecid = recidiverend: remissie (8.82/2.63)
| werk = betaald werk
| | BDI_COG.1 <= 10
| | | burgstaat = partner verloren door scheiding/overlijden: remissie (0.0)
| | | burgstaat = samenwonend/gehuwd
| | | | deprrecid = enkelvoudig: geen remissie (2.5/0.12)
| | | | deprrecid = recidiverend
| | | | | MADRS.1 <= 25: remissie (4.78/1.0)
| | | | | MADRS.1 > 25: geen remissie (2.23)
| | | burgstaat = niet samenwonend/gehuwd: remissie (14.54/3.36)
| | BDI_COG.1 > 10: geen remissie (10.77/1.23)

References

﻿Posternak MA, Miller I (2001). "Untreated short-term course of major depression: A meta-analysis of outcomes from studies using wait-list control groups". Journal of Affective Disorders 66 (2–3): 139–46.

﻿Major depressive disorder. (2008, December 6). In Wikipedia, The Free Encyclopedia. Retrieved 14:31, December 6, 2008, from http://en.wikipedia.org/w/index.php?title=Major_depressive_disorder&oldid=256226847