Discussion: View Thread

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I'll defer to others in the section about how best to fit the model with or without outliers. There are a set of regression diagnostics statistics in SAS, and R , especially in, Frank Harrell's HMISC library there are   tools,  for estimating "leverage" and lots of graphing options and so forth. Possibly a transformation of some variables may be in order.

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

Hey Georgette,

It sounds like the scientists or lab tech knows *why* these observations are outliers.

Are they all outliers for the same reason? What is this reason? Ex. is the reason more like a treatment/factor? Or is the reason on a continuum? 

Which "popular selection method" was used?

I'll defer to others in the section about how best to fit the model with or without outliers. There are a set of regression diagnostics statistics in SAS, and R , especially in, Frank Harrell's HMISC library there are   tools,  for estimating "leverage" and lots of graphing options and so forth. Possibly a transformation of some variables may be in order.

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

As someone who has worked in industry and someone who saw the damage removing "outliers" has on people, I'd keep the data in the model. 

At one production facility I worked at, the engineers used a 10% truncated Mean and Std Dev for their data. The Mean of this data was never really severely impacted by the truncation. The Std Dev was ALWAYS impacted by this AND would cause a lot of issues once we sold product to customers. For example, we would have a truncated mean and std of say Mean = 32units, Std Dev = 4 units. The customer would reject our supply if over 5% of their samples failed to meet a 25 unit threshold. According to our results, this occurs 4% of the time. 

The customer DOES NOT use a truncated data. They would find the mean to be say 32 units and the std dev to be 7 units. Based upon the full data, our product fails about 16% of the time. So, they reject our batch. 

If you look at the water quality data from Flint, Mi from a few years ago, the reason the water crisis went on for such a long time was, the scientists and engineers decided that a couple of the "high" samples were invalid because they are "Too Much" of an outlier.... based upon their (ignorant) assumptions. Had someone competent done the analysis, they would have used an appropriate method to test the results an found that there was NO OUTLIER! 

I've also worked with chemical analysis systems. We had what is called a raising baseline.  Meaning, if we measured the same thing 20 times in a row, we could plot the reported concentration of the analyte as a function of where it sat in the run sequence. We might get something like: % Recovery = 68% + 3%*Run Position. So, the 40th sample run, will have a reported % recovery of 188%. Clearly an outlier! By using the model I created for the data, we can create a correction factor for that data and report back a more "true" concentration of the analytes no matter what position they are in the run sequence. 

Hey Georgette,

It sounds like the scientists or lab tech knows *why* these observations are outliers.

Are they all outliers for the same reason? What is this reason? Ex. is the reason more like a treatment/factor? Or is the reason on a continuum? 

Which "popular selection method" was used?

I'll defer to others in the section about how best to fit the model with or without outliers. There are a set of regression diagnostics statistics in SAS, and R , especially in, Frank Harrell's HMISC library there are   tools,  for estimating "leverage" and lots of graphing options and so forth. Possibly a transformation of some variables may be in order.

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

It depends partly on the nature of the outliers, but as a general proposition, including an outlier indicator as an explanatory variable in a regression is problematic, especially with a relatively small dataset.

At the very least, creating new variables in a model based on an outlier assessment derived from the very same data will result in suspect p-values.  No standard p-value calculation accommodates this complicated interaction between variable creation and stochastic modeling of the responses.

When the outliers are outlying residuals, the basic issue is that the conditional distribution of the outlying data is unlikely to be anything like the conditional distribution of the remaining data.  Often it would have a greater variance, for instance, especially when the outliers are in both directions: outliers, by their nature, vary a lot!  Thus, in addition to incorporating a new explanatory variable, you will have to fit a model with heteroscedastic responses: not great when the observation : explanatory variable ratio is already down to just 10:1.

A better approach is to fit a robust regression (essentially least squares fit with IWLS).  But it sounds like it won't tell you something you don't already know: these data are indeed outliers and they do affect the fit.  One advantage of the robust regression, though, is that you can tell your client that the outlier identification is principled and not entirely up to your judgment.  Another is that it is susceptible to easy bootstrapping, simulation, and cross-validation, because the outlier identification and model fitting are performed in a single operation.  This won't help you here, with such a relatively small dataset, but in other circumstances it's well worth considering.

When the outliers are geometric outliers or (almost equivalently) outlying values of the explanatory variables, the situation is more delicate, because they likely are highly influential points.  Since you get a "very different model" depending on their inclusion, some of them must also be high-leverage points.  This is clear evidence that your conclusions are sensitive to this small subset of the data.  That might be the most important message (and even the most definite message) you can convey.  Your engineers will need to think hard about how their judgment concerning a small subset of the data, if incorporated in your modeling, determines the outcome.

(In my experience with engineers, which now spans almost 40 years, they are very good at coming up with plausible post hoc explanations for anything.  I respect that, because frequently those explanations provide insight, but I have been around long enough to know that our seemingly nit-picking "theoretical" concerns about data analysis or statistical modeling truly are valid in the real world.  Repeated opportunities to explain away unusual or unexpected data all too easily turn into a self-serving feedback loop that prevents anyone from learning anything about the system or phenomenon being studied.)

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?  

I have a stats question and appreciate the skill level of this group.  I have a regression analysis with let's say 1 continuous response and 3 continuous predictors from 30 observations-a mix of pre-set DOE levels and some other lab data at central values.  This final result is obtained after a popular selection method is used.   Using commonly accepted criteria, 3 observations are assessed as outliers and influencers.  Removing the outliers and fitting a regression of 27 rows yields a very different model. 

In a good classical way, the rationale for the outliers is assessed. The engineer provides a lab operations reason, not in the original file, for these 3 observations. Is it good statistical practice to create an indicator variable that identifies the 3 outliers, add it to the 'x' columns and refit the model?