Discussion: View Thread

The discussion on p-value interpretation has been interesting and turned into a lot more than I expected when I read Michael Chernick's initial post. I don't have anything more to add about p-values, and thanks to everyone who posted. I did want to share an interesting coincidence. Hugh Peach's post on Sunday referred to a 1971 Tversky and Kahneman paper. Earlier that day I was watching Fareed Zakaria's show on CNN. His guest was the same Daniel Kahneman, a psychologist and Nobel Prize recipient (for economics in 2002, shared with Tversky), and the discussion was on his new book, Thinking, Fast and Slow. (See http://www.nytimes.com/2011/11/27/books/review/thinking-fast-and-slow-by-daniel-kahneman-book-review.html?pagewanted=all.) For everyone's amusement, here's a question Kahneman has used in lectures and Sunday on CNN: A bat and a ball together cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost?

The discussion on p-value interpretation has been interesting and turned into a lot more than I expected when I read Michael Chernick's initial post. I don't have anything more to add about p-values, and thanks to everyone who posted. I did want to share an interesting coincidence. Hugh Peach's post on Sunday referred to a 1971 Tversky and Kahneman paper. Earlier that day I was watching Fareed Zakaria's show on CNN. His guest was the same Daniel Kahneman, a psychologist and Nobel Prize recipient (for economics in 2002, shared with Tversky), and the discussion was on his new book, Thinking, Fast and Slow. (See http://www.nytimes.com/2011/11/27/books/review/thinking-fast-and-slow-by-daniel-kahneman-book-review.html?pagewanted=all.) For everyone's amusement, here's a question Kahneman has used in lectures and Sunday on CNN: A bat and a ball together cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost?

The discussion on p-value interpretation has been interesting and turned into a lot more than I expected when I read Michael Chernick's initial post. I don't have anything more to add about p-values, and thanks to everyone who posted. I did want to share an interesting coincidence. Hugh Peach's post on Sunday referred to a 1971 Tversky and Kahneman paper. Earlier that day I was watching Fareed Zakaria's show on CNN. His guest was the same Daniel Kahneman, a psychologist and Nobel Prize recipient (for economics in 2002, shared with Tversky), and the discussion was on his new book, Thinking, Fast and Slow. (See http://www.nytimes.com/2011/11/27/books/review/thinking-fast-and-slow-by-daniel-kahneman-book-review.html?pagewanted=all.) For everyone's amusement, here's a question Kahneman has used in lectures and Sunday on CNN: A bat and a ball together cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost?

The discussion on p-value interpretation has been interesting and turned into a lot more than I expected when I read Michael Chernick's initial post. I don't have anything more to add about p-values, and thanks to everyone who posted. I did want to share an interesting coincidence. Hugh Peach's post on Sunday referred to a 1971 Tversky and Kahneman paper. Earlier that day I was watching Fareed Zakaria's show on CNN. His guest was the same Daniel Kahneman, a psychologist and Nobel Prize recipient (for economics in 2002, shared with Tversky), and the discussion was on his new book, Thinking, Fast and Slow. (See http://www.nytimes.com/2011/11/27/books/review/thinking-fast-and-slow-by-daniel-kahneman-book-review.html?pagewanted=all.) For everyone's amusement, here's a question Kahneman has used in lectures and Sunday on CNN: A bat and a ball together cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost?

The discussion on p-value interpretation has been interesting and turned into a lot more than I expected when I read Michael Chernick's initial post. I don't have anything more to add about p-values, and thanks to everyone who posted. I did want to share an interesting coincidence. Hugh Peach's post on Sunday referred to a 1971 Tversky and Kahneman paper. Earlier that day I was watching Fareed Zakaria's show on CNN. His guest was the same Daniel Kahneman, a psychologist and Nobel Prize recipient (for economics in 2002, shared with Tversky), and the discussion was on his new book, Thinking, Fast and Slow. (See http://www.nytimes.com/2011/11/27/books/review/thinking-fast-and-slow-by-daniel-kahneman-book-review.html?pagewanted=all.) For everyone's amusement, here's a question Kahneman has used in lectures and Sunday on CNN: A bat and a ball together cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost?

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

a* = Prob{H0 true ' p=a) = a(1-?)/[ a(1-?) + (1-ß) ?] where p= the p-value a = type I error ß = power at the specified alternative and ? = apriori probability that the null hypothesis is false.

Also ß* = Prob(H0 false ' p>a) = ß?/[ ß? +(1 - a)(1 - ?).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).

I'd like to add a new wrinkle to this discussion of p-values.  It is interesting that a barely significant or barely nonsignificant p-value is taken as a credible indication of statistical significance or nonsignificance.  Please examine these simple results and see what I mean.

I had a set of real data comparing the length of spinal fusion in two well-defined groups. Group 1 had n=20 and Group 2 had n=23, so the sample size seemed to be large enough that any conclusion would seem plausible.  The t-test p = 0.0446.  I got curious as to what would have happened if one patient had not shown up for the study. Using round-robin deletion, the resulting p-values ranged from 0.032 to 0.082, with 72% if p-values > 0.05.  So, if one patient was lost to follow-up, 72% of the possible outcomes would have suggested no significant difference, or an inability to say which group had the higher mean length.

FYI: If any two were lost to follow-up, the p-values would have ranged from 0.005 to 0.16. 

To see if a non-parametric method (Kruskal-Wallis) would not have this problem, the complete data set had p=0.036; excluding one record, the p-values ranged from 0.019 to 0.055; excluding two records, the p-values ranged from 0.006 to 0.085.  While the K-W test had less of a problem, a borderline significant p-value is still sensitive to who is and is not included in the analysis.

So, we see that conclusions based on marginally significant or non-significant p-values might simply depend on whether one or two patients were not lost to follow-up; I call this "fragile findings of statistical significance."  This is rarely mentioned in textbooks, where the computed p-value is the final story.

If we go back to Fisher, he thought that a single significant result was not the end of the story.  He believed that repetitions of the study with the same conclusion would provide the basis for having faith in the reported finding. The Neyman-Pearson notion of Type I and Type II error rates, with a = 0.05, have encouraged the use of one-shot studies (plus the time and cost of doing so).