Discussion: View Thread

Hello everyone,

I am doing some modeling for a client (linear regression) and they are insistent on being able to evaluate the model based on accuracy.  I've tried talking to them about the need to measure the model's performance based on RMSE, MSE, etc but they want to see its 'accuracy' expressed as a percentage.  Attempts to explain the difference between how we evaluate classification and regression models haven't been successful, yet.  Any advice from those who might have dealt with a similar issue?

Hello everyone,

I am doing some modeling for a client (linear regression) and they are insistent on being able to evaluate the model based on accuracy.  I've tried talking to them about the need to measure the model's performance based on RMSE, MSE, etc but they want to see its 'accuracy' expressed as a percentage.  Attempts to explain the difference between how we evaluate classification and regression models haven't been successful, yet.  Any advice from those who might have dealt with a similar issue?

How much data do you have? 

A previous employer demanded to know the accuracy of a regression model. I told them the R^2 value is 0.99978, or whatever it was. That was what they wanted. 

Something I did out of curiosity was to create  model with training data. Then look at how many of the testing data points for Y fit within +/- 1 SD, 2SD, etc. or within some relevant parameters say Y_hat +/- 5%, 10% etc. 

Hello everyone,

I am doing some modeling for a client (linear regression) and they are insistent on being able to evaluate the model based on accuracy.  I've tried talking to them about the need to measure the model's performance based on RMSE, MSE, etc but they want to see its 'accuracy' expressed as a percentage.  Attempts to explain the difference between how we evaluate classification and regression models haven't been successful, yet.  Any advice from those who might have dealt with a similar issue?

As a quick second on Andrew's comment with a very similar example to his (& apologies in advance for the self-citation), here's an example where....

• a Gaussian process regression model* is constantly being re-fit to new data using maximum a posteriori (MAP)
• but the "ultimate performance" of the model isn't the data with respect to the posterior predicted distribution....it's the lower quantiles of this performance.
• In other words, for our clinical situation, we don't care if the model is very accurate & precise most of the time (on the easy patients), we want to reduce the severity of the very worst predictions.
• So we developed an algo to identify models that optimize this analytical metric (in real time).

Link to Downloadable PDF: https://ieeexplore.ieee.org/document/8226743

Like the value of Andrew's follow-up analysis. Frequently the important part of a model is what it does when it get's things wrong, not how well it gets most thing right. That why it's important to not conflate "model performance metric" with "customer/user/business value".

* If you're not familiar with GP regression, just think of it as linear regression over the kernel space...plop a kernels on each of the data points & regress over that so you can estimate non-linear functions without (m)any icky assumptions.

How much data do you have? 

A previous employer demanded to know the accuracy of a regression model. I told them the R^2 value is 0.99978, or whatever it was. That was what they wanted. 

Something I did out of curiosity was to create  model with training data. Then look at how many of the testing data points for Y fit within +/- 1 SD, 2SD, etc. or within some relevant parameters say Y_hat +/- 5%, 10% etc. 

Hello everyone,

I am doing some modeling for a client (linear regression) and they are insistent on being able to evaluate the model based on accuracy.  I've tried talking to them about the need to measure the model's performance based on RMSE, MSE, etc but they want to see its 'accuracy' expressed as a percentage.  Attempts to explain the difference between how we evaluate classification and regression models haven't been successful, yet.  Any advice from those who might have dealt with a similar issue?

As a quick second to Glen's quick second ;-) I feel--heck, I KNOW this is not talked about enough in the applied world. I would personally like to see more formal coverage on the topic in applied stats courses--doesn't have to be extensive, just enough to make one think. There are many different ways to measure the "success" of a model, or rather, how you measure the "success" of the model depends on the practical objective of the model. In the end, what ultimately matters is how the decisions based on the model impact things that are real. Some of those metrics may not be "statistical" at all, so our responsibility becomes ensuring the model itself is "statistically sound" while maximizing whatever the objective function of interest....

As a quick second on Andrew's comment with a very similar example to his (& apologies in advance for the self-citation), here's an example where....

• a Gaussian process regression model* is constantly being re-fit to new data using maximum a posteriori (MAP)
• but the "ultimate performance" of the model isn't the data with respect to the posterior predicted distribution....it's the lower quantiles of this performance.
• In other words, for our clinical situation, we don't care if the model is very accurate & precise most of the time (on the easy patients), we want to reduce the severity of the very worst predictions.
• So we developed an algo to identify models that optimize this analytical metric (in real time).

Link to Downloadable PDF: https://ieeexplore.ieee.org/document/8226743

Like the value of Andrew's follow-up analysis. Frequently the important part of a model is what it does when it get's things wrong, not how well it gets most thing right. That why it's important to not conflate "model performance metric" with "customer/user/business value".

* If you're not familiar with GP regression, just think of it as linear regression over the kernel space...plop a kernels on each of the data points & regress over that so you can estimate non-linear functions without (m)any icky assumptions.

How much data do you have? 

A previous employer demanded to know the accuracy of a regression model. I told them the R^2 value is 0.99978, or whatever it was. That was what they wanted. 

Something I did out of curiosity was to create  model with training data. Then look at how many of the testing data points for Y fit within +/- 1 SD, 2SD, etc. or within some relevant parameters say Y_hat +/- 5%, 10% etc. 

Hello everyone,

I am doing some modeling for a client (linear regression) and they are insistent on being able to evaluate the model based on accuracy.  I've tried talking to them about the need to measure the model's performance based on RMSE, MSE, etc but they want to see its 'accuracy' expressed as a percentage.  Attempts to explain the difference between how we evaluate classification and regression models haven't been successful, yet.  Any advice from those who might have dealt with a similar issue?

To expiate my sin of self-reference, here is another example of medical research that's grappling with the fact that the model is optimized for an *analytically-tractable* metric that's less important than an alternative metric (that's less analytically tractable).

In this case it's (i) a logistic regression model with normal MLE training, versus (ii) the more useful metric is AUROC.

Here the article link: Developing biomarker combinations in multicenter studies via direct maximization and penalization - PubMed

Related work by Allison Meisner and others:

• https://pubmed.ncbi.nlm.nih.gov/29344362/
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5499057/

As a quick second to Glen's quick second ;-) I feel--heck, I KNOW this is not talked about enough in the applied world. I would personally like to see more formal coverage on the topic in applied stats courses--doesn't have to be extensive, just enough to make one think. There are many different ways to measure the "success" of a model, or rather, how you measure the "success" of the model depends on the practical objective of the model. In the end, what ultimately matters is how the decisions based on the model impact things that are real. Some of those metrics may not be "statistical" at all, so our responsibility becomes ensuring the model itself is "statistically sound" while maximizing whatever the objective function of interest....

As a quick second on Andrew's comment with a very similar example to his (& apologies in advance for the self-citation), here's an example where....

• a Gaussian process regression model* is constantly being re-fit to new data using maximum a posteriori (MAP)
• but the "ultimate performance" of the model isn't the data with respect to the posterior predicted distribution....it's the lower quantiles of this performance.
• In other words, for our clinical situation, we don't care if the model is very accurate & precise most of the time (on the easy patients), we want to reduce the severity of the very worst predictions.
• So we developed an algo to identify models that optimize this analytical metric (in real time).

Link to Downloadable PDF: https://ieeexplore.ieee.org/document/8226743

Like the value of Andrew's follow-up analysis. Frequently the important part of a model is what it does when it get's things wrong, not how well it gets most thing right. That why it's important to not conflate "model performance metric" with "customer/user/business value".

* If you're not familiar with GP regression, just think of it as linear regression over the kernel space...plop a kernels on each of the data points & regress over that so you can estimate non-linear functions without (m)any icky assumptions.

How much data do you have? 

A previous employer demanded to know the accuracy of a regression model. I told them the R^2 value is 0.99978, or whatever it was. That was what they wanted. 

Something I did out of curiosity was to create  model with training data. Then look at how many of the testing data points for Y fit within +/- 1 SD, 2SD, etc. or within some relevant parameters say Y_hat +/- 5%, 10% etc. 

Hello everyone,

I am doing some modeling for a client (linear regression) and they are insistent on being able to evaluate the model based on accuracy.  I've tried talking to them about the need to measure the model's performance based on RMSE, MSE, etc but they want to see its 'accuracy' expressed as a percentage.  Attempts to explain the difference between how we evaluate classification and regression models haven't been successful, yet.  Any advice from those who might have dealt with a similar issue?

Hello everyone,

I am doing some modeling for a client (linear regression) and they are insistent on being able to evaluate the model based on accuracy.  I've tried talking to them about the need to measure the model's performance based on RMSE, MSE, etc but they want to see its 'accuracy' expressed as a percentage.  Attempts to explain the difference between how we evaluate classification and regression models haven't been successful, yet.  Any advice from those who might have dealt with a similar issue?