Discussion: View Thread

  • 1.  DFA prior probabilities

    Posted 05-11-2011 16:38
    Can anyone tell me what the effect of assigning prior probabilities for group membership has on classification accuracy in a discriminant function analysis (DFA)?

    The goal of the analysis in question is evaluate the degree to which the pattern of scores on several variables differentiate between participants in 3 groups representing low, moderate and high levels of skill using DFA. I have assigned prior probabilities of .16, .63, and .21 based on the sample N for each group, assuming that these rates are representative of base rates for the 3 groups in the population. The issue is that I get radically different rates for the proportion correctly classified into the three groups when I assign prior probabilities compared to when I do not. When I assign prior probabilities, the proportion correct classified is .14, .88, and .21, for the 3 groups respectively. When I assume equal prior probabilities, the proportion correct classified is .86, .47, and .83, for the 3 groups respectively. There's a clear difference depending upon the decision to assign prior probabilities, but I'm not sure why, and I can't figure out which way I should go with this model. Any advice would be greatly appreciated. Thanks!

    -------------------------------------------
    Jennifer Rose
    Research Associate Professor
    Wesleyan University
    -------------------------------------------


  • 2.  RE:DFA prior probabilities

    Posted 05-11-2011 17:10

    You haven't told us what method you used .  It is probably Fisher's LDA.  But it doesn't really matter.  Just think about the Bayes Rule and assume you are classifying based on the Bayes Rule.  Side Note:  If the three classes have multivariate Gaussian distributions all with the same covariance matrix the Bayes Rule is LDA.

    The Bayes rule chooses the class with the largest posterior probability.

    Remember posterior = prior x likelihood.  So unless you have a large amount of data the prior has an effect on the aposteriori probability.  In effect you are giving more weight to the make an error in the class with the highest prior probability.  If you want a frequentist type analysis use equal priors 1/3 1/3 1/3 and then the likelihood i.e data determines the classification without favoring any of the classes.  You see what you did was put a lot more weight on class 2 so your rule did a good job correctly classifying cases from class 2 at the expense of 1 and 3.  When you used equal weights you were able to do quite well with classes 1 and 3 but not so well with class 2.  So apparently your training data separate classes 1 and 3 well but probably 1 and 3 overlap quite a bit with class 2. So equal weights tell you what the data says.  It makes perfect sense because in order to get good performance with class 2 you are forced to make a lot of mistakes with the overlapping cases from classes 1 and 3.

     

    How large is N?  For very large N you won't see such great sensitivity to the prior. Now if you had good reason to believe that the priors that you based on the sample proportion really do represent the population you are sampling from using them would be a good thing to do.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------








  • 3.  RE:DFA prior probabilities

    Posted 05-11-2011 17:22
    In several kinds of software the  prior probabilities are "equal" or "size".  With "equal" the goal it to assign the same number of cases to each group. With "size" the goal is to assign cases to groups according to the original size of the group. 
    If one has 3 groups with 10, 20, and 30 cases, the software tries to assign 20,20,20 when "equal" is specified.  So when crosstabbing the original group by the assigned group there have to be cases on the off-diagonal of the table.  Off-diagonal cells are "incorrect" assignments. 

    However, when the size of the assigned groups is the same as the size of the original groups, it is at least possible that all of off-diagonal cells be empty.

    A thought experiment.
    take 10 red chips, 20 white chips, and 30 blue chips. Put them in a bag. Shake the bag.  Put the first 20 draws in pile 1, the next 20, in pile 2, and the last 20 in pile 3. Call pile 1 red, pile 2 white, pile3 blue. 
    Draw a 3 by 3 table. Label the row direction actual color, and the column direction assigned color. call the row value labels red, white, blue.  The same with the column labels.  Place the chips in the cells.
    count how many are correct.  Of necessity you have at least 20 chips in the off diagonal.

    Do it again.  Except, put 10,20,30 in the 3 piles. Call  the first pile "red", the next pile "white", the third "blue". Place the pile in the 9 cells. 

    However, I do not know of a DFA program that takes into account that the levels of skill are ordered.

    How did you measure skill in the first place?

    If you do not have more fine-grained skill scores consider using Categorical Regression or ther software that takes into account that the skill levels are ordered.

    Did you try drawing a profile graph of the raw variables?  What does the scatterplot of the cases look like in the space defined by the functions?

    -------------------------------------------
    Arthur Kendall
    Social Research Consultants
    -------------------------------------------








  • 4.  RE:DFA prior probabilities

    Posted 05-11-2011 18:06
    I have a couple of questions:

    1. Your prior probabilities of .16, .63, and .21 based on sample N for each group: (a) How big was the total N of all three groups? (b) In choosing your sample, did you sample unrestrictedly? or did you use a procedure akin to frequency matching, in which one tries to force one's sample proportions to be equal to pre-determined population proportions?

    2. Will your classifier have an immediate practical use? or is this still in the proof-of-principle stage? 

    -------------------------------------------
    Eric Siegel
    Boistatistician
    Univ of Arkansas for Medical Sciences
    -------------------------------------------








  • 5.  RE:DFA prior probabilities

    Posted 05-12-2011 09:25

    Thanks for all the feedback. It has been extremely helpful!

    -------------------------------------------
    Jennifer Rose
    Research Associate Professor
    Wesleyan University
    -------------------------------------------