Discussion: View Thread

Anyone have any good references for comparing metrics, say like performance metrics, when the metrics may be based on different variables from the same data set?
Cheers,
Rhonda

-------------------------------------------
Rhonda Rosychuk
Univ of Alberta
-------------------------------------------

Perhaps the best method to compare 'apples' and 'oranges' is the signal to noise ratio or the non-centrality parameter.  To compare if a person is taller than he is smarter, one compute z{height} vs z{IQ}.  To compare means performance metrics, one would do something similar.  For example, to determine which treatment differences have greater performance one could compute (Mean{Active, height] - Mean{Control, height})/s.d.{height} and compare this 'standardized mean difference' with other parameters, like IQ.  Like the z-score, it presents how many s.d., the treatment differences are apart.  It a) has the ability to be understood by the non-expert (where mean differences might not be as easily understood), b) can be used directly in power computations, c) is unit free, d) can be computed very easily, e) can be used compare parameters across different studies, and f) can have its confidence band estimated.

-------------------------------------------
Allen Fleishman
Allen Fleishman Biostatistics Inc.
-------------------------------------------

Rhonda, can you be more specific about your objective?  Your question reminds me of the Derringer approach to finding a good balance between competing objectives (eg, tire treadwear, rolling resistance, and the various types of traction) but you may have something entirely different in mind.

Allen, your approach is interesting but does not consider importance to whatever the objective is.  If I'm recruiting for track and field I don't care about the candidate's musical ability (as compared to other candidates).  If I'm recruiting for a symphony orchestra I don't care whether the candidate can run.
-------------------------------------------
Emil M Friedman, PhD
emil.friedman@alum.mit.edu (forwards to day job)
emilfrie@alumni.princeton.edu (home)
http://www.statisticalconsulting.org
-------------------------------------------







Original Message:
Sent: 07-19-2012 11:12
From: Allen Fleishman
Subject: Comparing Metrics

Perhaps the best method to compare 'apples' and 'oranges' is the signal to noise ratio or the non-centrality parameter.  To compare if a person is taller than he is smarter, one compute z{height} vs z{IQ}.  To compare means performance metrics, one would do something similar.  For example, to determine which treatment differences have greater performance one could compute (Mean{Active, height] - Mean{Control, height})/s.d.{height} and compare this 'standardized mean difference' with other parameters, like IQ.  Like the z-score, it presents how many s.d., the treatment differences are apart.  It a) has the ability to be understood by the non-expert (where mean differences might not be as easily understood), b) can be used directly in power computations, c) is unit free, d) can be computed very easily, e) can be used compare parameters across different studies, and f) can have its confidence band estimated.

-------------------------------------------
Allen Fleishman
Allen Fleishman Biostatistics Inc.
-------------------------------------------

The original thread author just asked about comparing metrics.  Yes, Ms. Rosychuk didn't explain the problem.  And I heartily agree that many problems that I've been given become quite different when I fully understand what the issues are.  [My very first statistical analyst job (I was a senior in college at the time), the client (a PhD student) asked me to force a two way balanced design analysis to complete despite highly correlated d.v. with negative sum of squares estimates.  After the job was finished they told me they wanted to do a factor analysis, but couldn't due to small N, so they did all pairwise ANOVAs instead.] 

Without knowing any more I answered Rhonda's questions about how to compare metrics.  Nothing more.  WRT irrelevant parameters (musical or political acumen), that is only relevant to the client.  If they want to select runners based on their musical ability, that's their problem.  If they are examining tertiary parameters and find that some irrelevant parameters might be better than their primary, then they might need to re-assess their entire study and results.  Again something not relevant to a forum on statistical issues.  It is the job of statisticians to examine parameters for utility.  I examined data for an orphan drug where a major d.v. was physical ability to move.  The key parameter in the study was 4 minute walking distance.  A secondary was time to climb stairs.  I observed that the stair climbing had an effect size almost twice that of 4 minute walking time.  I recommended the client reconsider the stair climbing.  I recommended that they should do a literature review of other studies and determine if other parameters might be better.  My client, a pharmaceuticals company, did not include any musical tests in their battery.  I would guess few would.

With regard to your Derringer approach, the different parameters you included are (to me) obviously different and often competing measures of tires.  That is, tread wear and traction are likely to be negatively correlated.  On the other hand, there may be wear at the medial aspect of the tire, wear on the edge, mileage to 1/4 inch of tread, etc.  Selection among these different tread wear parameters might be useful.

Sometimes answering specific questions is better than smoke about musical vs political acumen.

-------------------------------------------
Allen Fleishman
Allen Fleishman Biostatistics Inc.
-------------------------------------------