ASA Connect

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

I appreciate the move to start a foundational discussion in this forum! And glad to see a topic that I often think about. My background is also in physics (now in collaborative statistics) so these quantum mechanical metaphors have always landed home for me.

I generally agree that outcomes are not (cannot?) be independent of measurement. And the points about the methods of measurement influencing the outcome are well taken. 

I'd just like to add that if we take a step even further back, and consider that what outcomes we decide to measure and how they are conceptualized (i.e. how we operationalize our constructs) are also part of the measurement process. And from this view, it seems to me that the outcomes are not only conditional on the method of measurement, but also the worldview of the researcher (as mediated through the choices / conceptualization of what the outcomes even are).

 The angle or worldview or value system from which we choose to view a phenomenon (i.e. process) affects what we see as worth measuring and therefore affects what and how we choose to measure it. 

Through validity testing we can make sure our measures are reasonably stable, reliable, etc., and therefore we do not completely err in our conceptualizations. But there is no guarantee that it is the only valid view of the phenomenon. 

At best we can get agreement that given our values and priorities, our view is good enough for the inferences we would like to make. But there is always the chance that an outcome that appeared valid is rendered invalid by a reevaluation of what really matters. I think that health disparities research is a good example of this. If our outcomes ignore the possibility of disparities, we could see one set of results (for example, on average positive outcomes). And if the issue of disparities is included in our conceptualization of the healthcare process, we could get a different set of results (maybe outcomes are only positive for some groups). 

Although there is nothing statistically incompatible between these two sets of results, the inferences (and decisions) made from them could be radically different. So inference is not independent of our approach to conceptualization and measurement of phenomenon. I think this points to a (maybe more radical?) version of the observer-participancy that Jonathan mentioned. That the worldview and values of the observed is inextricably tied to the measurement process. And thus to the outcomes.

Alex E. Clain

Postdoctoral Fellow

Communication Sciences and Disorders

Northwestern University 

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

I appreciate the move to start a foundational discussion in this forum! And glad to see a topic that I often think about. My background is also in physics (now in collaborative statistics) so these quantum mechanical metaphors have always landed home for me.

I generally agree that outcomes are not (cannot?) be independent of measurement. And the points about the methods of measurement influencing the outcome are well taken. 

I'd just like to add that if we take a step even further back, and consider that what outcomes we decide to measure and how they are conceptualized (i.e. how we operationalize our constructs) are also part of the measurement process. And from this view, it seems to me that the outcomes are not only conditional on the method of measurement, but also the worldview of the researcher (as mediated through the choices / conceptualization of what the outcomes even are).

 The angle or worldview or value system from which we choose to view a phenomenon (i.e. process) affects what we see as worth measuring and therefore affects what and how we choose to measure it. 

Through validity testing we can make sure our measures are reasonably stable, reliable, etc., and therefore we do not completely err in our conceptualizations. But there is no guarantee that it is the only valid view of the phenomenon. 

At best we can get agreement that given our values and priorities, our view is good enough for the inferences we would like to make. But there is always the chance that an outcome that appeared valid is rendered invalid by a reevaluation of what really matters. I think that health disparities research is a good example of this. If our outcomes ignore the possibility of disparities, we could see one set of results (for example, on average positive outcomes). And if the issue of disparities is included in our conceptualization of the healthcare process, we could get a different set of results (maybe outcomes are only positive for some groups). 

Although there is nothing statistically incompatible between these two sets of results, the inferences (and decisions) made from them could be radically different. So inference is not independent of our approach to conceptualization and measurement of phenomenon. I think this points to a (maybe more radical?) version of the observer-participancy that Jonathan mentioned. That the worldview and values of the observed is inextricably tied to the measurement process. And thus to the outcomes.

Alex E. Clain

Postdoctoral Fellow

Communication Sciences and Disorders

Northwestern University 

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

Very happy to see statistical physicists weighing in on this important topic. As a statistical physicist myself, I would like to expand on the useful remarks made by David Marker. 

As Dr. Marker has described, interaction between the observer and the thing being measured introduces and element of randomness into the measurement.  There is a view in quantum mechanics, known as the Copenhagen Interpretation, that this randomness is intrinsic. This approach would argue that population outcomes do not exist until observed. As I am one of the few physicists who reject the Copenhagen Interpretation, my colleagues here may wish to reject my comments ab ovo. 

A slightly different view is that the population outcomes really do exist - we just don't know preciscely what they are due to human interaction with the thing being measured. Lack of knowledge about a thing is not an argument against ontology. This view regards randomness not as intrinsic but stil containing an element that is irreducible, even with the best measurements. 

For example: I presently work for an electrical utility, using phyisc-informed AI to convert weather forecasts into a prediction for the number of power outages in a given geographic area in a period of time. The fact that we don't know specifically which wires will break due to high winds does not mean it doesn't exist, as customer losing power will be glad to point out to us. However, there exists an irreducible uncertainty in the process that must remain even with the best measurements. 

Irreducible randomness is an important concept for us as statisticians. It's important to keep in mind that some of the irreducible randomness in the systems we study can be introduced by human interaction - for example, when a person asks a question in a survey or takes a medical measurement. However, some irreducible randomness occurs without any human interaction. Electrical wires will break in high winds with some degreee of randomness whether they are observed or not, and the act of observing them does not introduce additional randomness into our understanding of the outcome. As statisticians, our interest is in the irreducible randomness whether introduced by human observation or not. 

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach.  Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer.  Hawthorne-type effects may change the behavior of patients in a study or clinical trial.  Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years.  But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.