ASA Connect

 View Only

  • 1.  diagnostic tests for multinomial logistic regression

    Posted 02-03-2015 16:13
    We are developing a model to evaluate credit risk and using multinomial logistic regression to estimate probability of default. The model validation is requiring us to run diagnostic tests to ensure that the multinom model is statistically valid. We are running

    Out-of-sample AUC/Gini to test for overfitting
    Likelihood-ratio Test for variable significance
    VIF for multicollinearity.

    We are asked to run additional tests. The items in our mind is stationarity and independence between observations.

    I want to ask what statistical tests are availalbe to test those two items for multinomial logistic regression?
    And is there additional properties that I need to test?

    thanks in advance
    -------------------------------------------
    Weizhong Li
    Senior Financial Institution Examiner
    Department of Business Oversight
    -------------------------------------------


  • 2.  RE: diagnostic tests for multinomial logistic regression

    Posted 02-04-2015 09:43

    Have you looked at SAS? 
    -------------------------------------------
    Aubrey Magoun
    Consultant
    Applied Research & Analysis, Inc.
    -------------------------------------------


    Original Message


  • 3.  RE: diagnostic tests for multinomial logistic regression

    Posted 02-04-2015 12:27

    I wish we are using SAS. Our previous modellers develop the multinomial model from R. He developed all the model fitting functions in R so we are not even using the open source R fitting function. We are using internal develop ones that does not have ability to spit anything other than the fitting coefficients.

    -------------------------------------------
    Weizhong Li
    Senior Financial Institution Examiner
    Department of Business Oversight
    -------------------------------------------


    Original Message


  • 4.  RE: diagnostic tests for multinomial logistic regression

    Posted 02-04-2015 09:51
    I'm assuming there is a time variable in your model since you are testing for stationarity? 

    -------------------------------------------
    Suzann Williams
    -------------------------------------------


    Original Message


  • 5.  RE: diagnostic tests for multinomial logistic regression

    Posted 02-04-2015 14:41

    there is but they are not in order. Each monthly record is associated with a time but we shuffle it so time is not in order. We also think stationarity is not meaningful in this context.
    -------------------------------------------
    Weizhong Li
    Senior Financial Institution Examiner
    Department of Business Oversight
    -------------------------------------------


    Original Message


  • 6.  RE: diagnostic tests for multinomial logistic regression

    Posted 02-04-2015 15:07
    I would worry less about the statistical test of significance and more about the business significance. Given the sample size of most of these models, almost any small deviation is significant. I have seen models that are virtually useless from a business standpoint, but very significant statistically. So you want to show that the model is much better than the current process, whatever that is.

    Is the OTS (out of time sample) representative of the environment into which the model is being implemented? The OTS performance should give the business an idea of the expected uncertainty in implementation.

    Are you more concerned about rank ordering than predictive accuracy (this is normal in credit risk)? If so then you want to look at the rank ordering compared to what is currently being done.

    Do you have a good monitoring plan in place so that any performance and stability issues can be identified as soon as possible.

    These are just a few of the issues that need to be addressed with this kind of model. You may notice that none of these issues are statistical, but more business issues.



    -------------------------------------------
    Michael Mout
    MIKS
    -------------------------------------------


    Original Message