Discussion: View Thread

Dear all,

 When we apply statistics it is to some substantive field. People working in or using the field have varying depths of knowledge about it. What level of knowledge should a statistician have of the field to be able to safely comment on the use of statistics there? Courts have to answer this question whenever they qualify a statistician to testify as an expert with regard to an analysis offered as evidence.  Organizations have to answer this question whenever they call upon a consulting statistician to comment on a new product development, a research finding, a survey, etc.

 The answer will vary with the circumstances, so let me pose a particular one, to ask what level of knowledge of the field you think is required. Assume the following:

• A statistician is asked to comment on the statistical merits of a report.
• The report is not an extraordinary piece of work that might call on the greatest minds or vast experience in the field.
• After some homework, the statistician can follow the points made, and in particular, can see to what effect statistical methods were applied-or failed to be applied when they should have been.
• The statistician has a graduate degree level of acquaintance with the methods involved (MS or PhD I don't care).

 This question comes up I suspect every time a statistical consultant enters a project. I would be greatly interested to get your comments.

-------------------------------------------
Will Fairley
Senior Statistician and President
Analysis & Inference, Inc.
-------------------------------------------

My sense is that the statistician must either know enough or be able to ask the right questions to determine whether the assumptions built into the analysis are followed closely enough that the calculated p-values and other results will be reasonably accurate.

The assumptions I'm thinking about are not typically normality assumptions, but independence assumptions.  For example, if there are replicates with treatment A and with treatment B, are the replicates independent?  Typically the answer is no. 

Often the replicates for treatment A will be on one set of raw materials and the replicates for treatment B will be on another and they will each be based on subsamples from a single batch.  In that case, differences could be from the treatments or from the different raw materials and the standard errors calculated from the replicates are likely to give a false sense of the precision of the comparison between the two treatments.

In my experience, there are many ways that confounding variables of one kind or another can find their way into comparisons and it requires persistence (in questioning the experts) and/or expertise on the part of the statistician to identify and control for those potential confounders. 

-------------------------------------------
Michael Morton
-------------------------------------------

Michael already answered what would have been most of my own response.

I would add that a precursor to deciding whether the assumptions underpinning
the analysis were met is to be absolutely sure you understand what was actually done
for the analysis; i.e., where did the numbers come from?  If you don't understand it,
then in my experience, it's at least as likely that the reason is that the report doesn't
explain clearly what was done than that the report just uses jargon you don't understand.

The ability to determine this may of course depend on the possibility of communicating
with the authors of the report if you have questions. 

-------------------------------------------
Katherine Godfrey
-------------------------------------------




Original Message:
Sent: 10-08-2012 11:27
From: Michael Morton
Subject: Knowlege required to comment on statistics in a substantive field

My sense is that the statistician must either know enough or be able to ask the right questions to determine whether the assumptions built into the analysis are followed closely enough that the calculated p-values and other results will be reasonably accurate.

The assumptions I'm thinking about are not typically normality assumptions, but independence assumptions.  For example, if there are replicates with treatment A and with treatment B, are the replicates independent?  Typically the answer is no. 

Often the replicates for treatment A will be on one set of raw materials and the replicates for treatment B will be on another and they will each be based on subsamples from a single batch.  In that case, differences could be from the treatments or from the different raw materials and the standard errors calculated from the replicates are likely to give a false sense of the precision of the comparison between the two treatments.

In my experience, there are many ways that confounding variables of one kind or another can find their way into comparisons and it requires persistence (in questioning the experts) and/or expertise on the part of the statistician to identify and control for those potential confounders. 

-------------------------------------------
Michael Morton
-------------------------------------------

Thank you to all those who responded to my question "RE:Knowledge required to comment on statistics in a substantive field." I'm glad that no one felt it necessary to point out that "knowledge" itself was mis-spelled in the original post!

For your interest, here, in brief, are excerpts or paraphrases of responses I got that were most apt--to me:

• The statistician must either know enough or be able to ask the right questions to determine whether the assumptions built into the analysis are followed closely enough that the calculated p-values and other results will be reasonably accurate; and is vigilant enough to seek out confounders that bedevil probably most comparisons.

• Surely, an answer to this question depends on how risk averse you are. Can you "safely comment" on something? You need to think about what are the consequences of giving bad advice and how likely it is that you will end up giving bad advice. You also have to evaluate what will happen if you decline to comment.

• I probably err on the side of commenting. Better to offer an opinion that your clients can accept or reject than to just shrug your shoulders. Most clients hate a statistician who is always qualifying their advice, who is always saying careful and cautious things, and who is unwilling to stick out his/her neck.

• The statistician should feel comfortable to review and provide comments on a report and feel that they have previous experience in the field they are reviewing (e.g., clinical trials in the diabetes area).

• A precursor to deciding whether the assumptions underpinning the analysis were met is to be absolutely sure you understand what was actually done for the analysis; i.e., where did the numbers come from? 

Best regards,

Will

-------------------------------------------
William Fairley
Senior Statistician and President
Analysis & Inference, Inc.
-------------------------------------------







Original Message:
Sent: 10-08-2012 18:06
From: Katherine Godfrey
Subject: Knowlege required to comment on statistics in a substantive field

Michael already answered what would have been most of my own response.

I would add that a precursor to deciding whether the assumptions underpinning
the analysis were met is to be absolutely sure you understand what was actually done
for the analysis; i.e., where did the numbers come from?  If you don't understand it,
then in my experience, it's at least as likely that the reason is that the report doesn't
explain clearly what was done than that the report just uses jargon you don't understand.

The ability to determine this may of course depend on the possibility of communicating
with the authors of the report if you have questions. 

-------------------------------------------
Katherine Godfrey
-------------------------------------------

Surely, an answer to this question depends on how risk averse you are. Can you "safely comment" on something? You need to think about what are the consequences of giving bad advice and how likely it is that you will end up giving bad advice. You also have to evaluate what will happen if you decline to comment.

There's a rather tragic example of giving bad statistical advice (though it was a doctor and not a statistician giving the advice) involving Sir Roy Meadow. I summarized some of this in Chance News #3
--> http://test.causeweb.org/wiki/chance/index.php/Chance_News_3
but you should also try an Internet search of "Meadow's Law".

When I consult with doctors, nurses, and other health care professionals, I try hard not to overstep my bounds. I tell them that I'm still trying to figure out the difference between good cholesterol and bad cholesterol. It's an exaggeration, of course, but it gets my point across. I also tell them that I am in a better position to ask questions than to answer them.

Even with this restraint, though, I probably err on the side of commenting. Better to offer an opinion that your clients can accept or reject than to just shrug your shoulders. Most clients hate a statistician who is always qualifying their advice, who is always saying careful and cautious things, and who is unwilling to stick out his/her neck. Maybe they SHOULD prefer the hedger and the qualifier, but they don't.

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------

The "Meadow" example wasn't a statistician having insufficient knowledge of medicine.  It was a pediatrician who didn't understand some pretty basic statistics.
-------------------------------------------
Emil M Friedman, PhD
emil.friedman@alum.mit.edu (forwards to day job)
emilfrie@alumni.princeton.edu (home)
http://www.statisticalconsulting.org
-------------------------------------------







Original Message:
Sent: 10-08-2012 11:41
From: Stephen Simon
Subject: Knowlege required to comment on statistics in a substantive field

Surely, an answer to this question depends on how risk averse you are. Can you "safely comment" on something? You need to think about what are the consequences of giving bad advice and how likely it is that you will end up giving bad advice. You also have to evaluate what will happen if you decline to comment.

There's a rather tragic example of giving bad statistical advice (though it was a doctor and not a statistician giving the advice) involving Sir Roy Meadow. I summarized some of this in Chance News #3
--> http://test.causeweb.org/wiki/chance/index.php/Chance_News_3
but you should also try an Internet search of "Meadow's Law".

When I consult with doctors, nurses, and other health care professionals, I try hard not to overstep my bounds. I tell them that I'm still trying to figure out the difference between good cholesterol and bad cholesterol. It's an exaggeration, of course, but it gets my point across. I also tell them that I am in a better position to ask questions than to answer them.

Even with this restraint, though, I probably err on the side of commenting. Better to offer an opinion that your clients can accept or reject than to just shrug your shoulders. Most clients hate a statistician who is always qualifying their advice, who is always saying careful and cautious things, and who is unwilling to stick out his/her neck. Maybe they SHOULD prefer the hedger and the qualifier, but they don't.

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------

Interesting question.  Personally as a statistical consultant I let new clients know my background and give them a copy of my CV and resume.  For example I have a lot of experience in the diabetes area and clinical trials in this area from Phase 1-4. I have had some requests to consult in clinical trials in the cancer area and I suggest a statistician that has a lot of experience in this area and also has a lot of survival analysis experience. Also if the client is looking for adaptive designs I suggest someone else.

I think clients looking for a consulting statistician are looking for someone with a depth of experience in selected areas and a good consultant statistician knows his or hers areas of expertise. So to answer your question the statistician should feel comfortable to review and provide comments on a report and feel that they have previous experience in the field they are reviewing. 

-------------------------------------------
Rocco Brunelle
Senior Statistician
Bowsher Brunelle Smith LLC
-------------------------------------------