ASA Connect

If the data set is large, then recursive partitioning is a good method. There is R code. SAS JMP is better. We studied various codes some years ago. The best version we found is commercial from Golden Helix.

JMP's stepwise platform is superb.  You can have it add or remove terms automatically, you can do it manually by clicking on boxes, you can have it stop at each step to let you override its decisions.  If have higher order terms, you can tell it to constrain your models to be hierarchical.  You can to lock certain terms in or out.  You can also do all possible regressions in the same platform.  You can save the history into a data table and plot model versus various different metrics (AIC, BIC, R-square, R-Square adjusted, Mallow's Cp, etc).

It has a creative default method for nominal variables that you can use or turn off.

Recursive partitioning is useful for exploratory analysis, as it can pick out a small subset of the variables that have good predictive power and it usually produces a comprehensible set of rules that can be easily encoded in SQL in an operational database.

There are downsides, though. First, it is very unstable - change the training set even slightly and you can get a very different tree. Second, it's predictive power is modest. If you want predictive power go for random forests, but that works as a black box, with no comprehensible model.

I would use it for EDA only.

Blaise 

The Flom and Cassel paper mentions the importance of the bias-variance tradeoff in considering the number of regressors kept, and the importance of "...eliminating models that do not make substantive sense."  An important factor in such 'sense,' considering the various ways variables may interact, and perhaps what they had in mind, is to consider subject matter theory.  Otherwise the number of combinations of variables to consider could become immense, not to mention consideration of nonlinearity.  In such a case, there could be numerous spurious results if all we look at are the data in a vacuum of subject matter theory.  Principal components would muddle such considerations, and other methods mentioned may also be too oriented toward data exploration - not that data exploration is not important, but here it needs help.  Perhaps it is generally better to just consider a range of reasonable models, and rely on the validation with independent data that would follow anyway.

The Flom paper succinctly presents many of the major concerns about step-based methods, which have a pronounced tendency to be opportunistic by exploiting idiosyncratic features of your data set that may not generalize to the population. Hence, model validation is strongly recommended.

Two other features of step methods sometimes overlooked are: (a) you have no guarantee that the best ensemble of independent variables will be selected; and (b) changing the enter/remove criteria can affect the efficacy of the procedure in finding a near-optimal ensemble, especially when there is multicollinearity and/or many IVs from which to build a model.

Elastic net is available in SPSS (and R) together with ridge regression and lasso. Tuning parameters lambda may be set by cross-validation or bootstrap. A review paper (Pavlou m, Ambler G, Seaman, S, De Iorio M, Omar RZ: Review and evaluation of penalized regression methods for risk prediction in low-dimensional data with few events. Statistics in Medicine 2014; 35: 1159-1177) outlines when which is best. Their favourite: Stochastic search variable selection, a Bayesian method using spike and slab priors.

Boosting should be considered.

If "10 events per variable" (Vittinghoff E, McCulloch CE: Relaxing the rule of ten events per variable in logistic and Cox regression. Am J Epidemiol. 2007; 165: 710-718) mandates small models, all of these could be estimated and the best one chosen by Mallow's Cp, AIC, BIC, ... . It depends, which objective has priority: Estimation, prediction, selection, mechanistic insight. Harrell F: Regression Modeling Strategies. Springer 2001 discusses this in detail.

Sas Proc GLMSelect can perform variable selection with LASSO, adaptive LASSO, or elastic net.

It can also do model averaging.  That's the procedure that I use as an alternative to outdated stepwise regression.

Brandy Sinco

MS Statistics, MA Mathematics

As always, there are some very good comments here.

Rather than recommending a specific technique in lieu of stepwise regression, I will merely observe that before choosing a technique one must be sure that they understand the desired purpose of the model to be built.  Many researchers plunge ahead with a "multivariable model" without a clear understanding of the model's purpose. 

For example, in some medical research papers, the primary objective is to determine whether a certain group of patients is at higher risk of some event (say, cardiovascular mortality) than those not in the group, in which case a multivariable model may be used to loosely "adjust" for some potential confounders.  In that case, we are less concerned about the model's overall fit and properties, and more concerned about whether the additional covariates in the model are those which would confound the primary relationship of interest - since the primary goal is to answer the question "Is (Group X) at higher risk of (Outcome Y)?"

In other medical research papers, the primary objective is to develop a comprehensive "risk score" - in which case the principal concern is ensuring strong model fit and good predictive accuracy, and the individual factors' effects may matter less than the overall model fit.  With the primary goal being accurate prediction of outcome, we must structure our modeling in such a way to achieve that goal (although reasonable people will disagree on how best to achieve that!)