Professor (Retired) and Adjunct Professor, Faculty of Business and Information Technology

Original Message:

Sent: 04-03-2024 14:07

From: Alex Clain

Subject: Discussion content: Do population outcomes exist independent of measurement?

I appreciate the move to start a foundational discussion in this forum! And glad to see a topic that I often think about. My background is also in physics (now in collaborative statistics) so these quantum mechanical metaphors have always landed home for me.

I generally agree that outcomes are not (cannot?) be independent of measurement. And the points about the methods of measurement influencing the outcome are well taken.

I'd just like to add that if we take a step even further back, and consider that what outcomes we decide to measure and how they are conceptualized (i.e. how we operationalize our constructs) are also part of the measurement process. And from this view, it seems to me that the outcomes are not only conditional on the method of measurement, but also the worldview of the researcher (as mediated through the choices / conceptualization of what the outcomes even are).

The angle or worldview or value system from which we choose to view a phenomenon (i.e. process) affects what we see as worth measuring and therefore affects what and how we choose to measure it.

Through validity testing we can make sure our measures are reasonably stable, reliable, etc., and therefore we do not completely err in our conceptualizations. But there is no guarantee that it is the only valid view of the phenomenon.

At best we can get agreement that given our values and priorities, our view is good enough for the inferences we would like to make. But there is always the chance that an outcome that appeared valid is rendered invalid by a reevaluation of what really matters. I think that health disparities research is a good example of this. If our outcomes ignore the possibility of disparities, we could see one set of results (for example, on average positive outcomes). And if the issue of disparities is included in our conceptualization of the healthcare process, we could get a different set of results (maybe outcomes are only positive for some groups).

Although there is nothing statistically incompatible between these two sets of results, the inferences (and decisions) made from them could be radically different. So inference is not independent of our approach to conceptualization and measurement of phenomenon. I think this points to a (maybe more radical?) version of the observer-participancy that Jonathan mentioned. That the worldview and values of the observed is inextricably tied to the measurement process. And thus to the outcomes.

Alex E. Clain

Postdoctoral Fellow

Communication Sciences and Disorders

Northwestern University

------------------------------

Alex Clain

PhD Candidate

Northwestern University

Original Message:

Sent: 04-02-2024 08:49

From: Jonathan Siegel

Subject: Discussion content: Do population outcomes exist independent of measurement?

It has been a while since this forum has had a serious discussion on a foundational topic. It's been mostly used to post announcements, get help on various topics, etc. So this post may now seem inappropriate for this forum, but I will essay it and see what people think.

In the long review cycle for our recent paper, we encountered the review comment "Population outcomes exist independent of measurement," in the course of objecting to some of our material. This is of course a key assumption of both classical and causal inference. But is it true?

One of W. Edwards Deming's differences with the statistical theory of the previous century was his rejection of exactly this assumption. It isn't true at all in quantum mechanics, where it's clear that measurement arises from an interaction between a measurement process and the measured. At this micro a scale, you can only measure something by throwing something at it, e.g. a photon, that which will interact with and alter the thing measured, making position and momentum, among other quantities, uncertain. And as a result, as Malley and Hornstein (Quantum statistical inference, Statist. Sci. 8(4): 433-457 (November, 1993). DOI: 10.1214/ss/1177010787) noted, inference is different. Hilbert spaces do not form a Boolean algebra. Joint distributions may not exist. Malley and Hornstein questioned whether frequentist theory has foundations in this context; they advocated a Bayesian approach. Myron Tribus, a leading Deming student of the last century, had reached a similar conclusion regarding the appropriate way to formalize Deming's approach.

Deming, a physicist by training who used quantum mechanics analogies in the technical parts of his arguments more generally, suggested that the outcome independence assumption isn't true in general. He posited that there is an issue of participant-observer interaction, interaction between measurement process and what is measured, in general, and particularly in complex systems such as biological ones, and indeed pretty much anything having to do with humans. He suggested that participant-observer effects may be rampant in social inquiry. He noted that, for example, people may give different answers to a survey depending on whether they get a male or a female interviewer. Hawthorne-type effects may change the behavior of patients in a study or clinical trial. Deming proposed an approach that does not assume any such independence. He emphasized operational definitions. Changing the method of measurement changes the outcome. He emphasized limiting statistical approaches to situations where the process has first been shown to behave approximately randomly, i.e. is in statistical control. And like Poincare before him, he emphasized qualitative approaches where quantitative ones have questionable validity or require too much computational complexity.

More fundamentally, Deming conceived of the kind of statistics that is actually useful in human affairs as generally requiring the study of dynamic processes, not static populations. The kind of questions people want to ask are generally analytical, not enumerative, in character, about predicting the future, not simply documenting the present. The conception of statistics as fundamentally being a science of processes, as distinct from being a science of data, has lost some of its former vogue in recent years. But one potential advantage of looking at things this way is that while static population outcomes have to be assumed to be independent of measurement, process outcomes do not.

Whether one agrees with this approach or not, one interesting observation about the process of having the paper reviewed was to notice that members of the statistics community today are still inclined to posit assumptions that make their particular inference theory work as being facts about the world. This, I suspect, comes from conceiving of statistics as being a branch of mathematics, which arrives at truths by starting with unshakably true foundational axioms and deriving theories by a process of deductive logic, rather than fundamentally a branch of science, which arrives at truths inductively, by a process of generalizing from observation, and whose principals can have no unshakably firm or certain foundations. If what we observe in the world is different from what theory assumes, it is the theory that has to bend.

------------------------------

Jonathan Siegel

Director Oncology Statistical Sciences

Bayer US Pharmaceuticals

------------------------------