ASA Connect

I have a collection of correlated r x c contingency tables.  The columns are time periods 1 to t.  In each contingency table, one column c* is a benchmark for statistical significance testing.   For any cell in a column c' not c*, the value in the cell is tested against the corresponding cell in c* for statistical significance.

In each time period, I collect a different and unique sample of size n according to a complex design.  The rows of the kth contingency table are values of a categorical variable that measures an important characteristic of the members of the sample in a given time period.

Since multiple characteristics of the sample members are represented across the contingency tables, the significance tests are not independent.  Should I control for multiple comparison risk by

(1) the Bonferroni correction,
(2) the Benjamini-Hochberg or Benjamini-Yekutieli correction,
(3) fit an unsaturated log linear model and declare significant cells in columns other than c* with extreme standardized residuals, or
(4) some other way? 

Thank you.

J. Conklin
Laurel, MD

------------------------------
Joseph Conklin
Senior Statistician
Office of Justice Programs
------------------------------

Hi Joseph,

You might investigate the work of Peter Westfall, who helped create SAS, PROC Multtest.

MIke Miller
Langhorne, PA

------------------------------
Michael Miller
Mfmillstat, Ltd.
------------------------------

Original Message:
Sent: 12-21-2021 10:47
From: Joseph Conklin
Subject: Strategies for Managing Multiple Comparison Risk in Correlated Contingency Tables

I have a collection of correlated r x c contingency tables.  The columns are time periods 1 to t.  In each contingency table, one column c* is a benchmark for statistical significance testing.   For any cell in a column c' not c*, the value in the cell is tested against the corresponding cell in c* for statistical significance.

In each time period, I collect a different and unique sample of size n according to a complex design.  The rows of the kth contingency table are values of a categorical variable that measures an important characteristic of the members of the sample in a given time period.

Since multiple characteristics of the sample members are represented across the contingency tables, the significance tests are not independent.  Should I control for multiple comparison risk by

(1) the Bonferroni correction,
(2) the Benjamini-Hochberg or Benjamini-Yekutieli correction,
(3) fit an unsaturated log linear model and declare significant cells in columns other than c* with extreme standardized residuals, or
(4) some other way?

Thank you.

J. Conklin
Laurel, MD

------------------------------
Joseph Conklin
Senior Statistician
Office of Justice Programs
------------------------------

Joseph,

To properly address your questions, which are sophisticated and complicated to answer, would require at least a better understanding of the structure of your experimental design and procedure, preferably without revealing any details of the variables or their meaning. Your sketch of the experiment and data structure situation is an excellent start, so if you're able to provide more detail to flesh out the underlying experimental questions while maintaining your confidentiality, you may just have come to the right group. Unfortunately, the timing of your question in this holiday season will no doubt limit the number of responses, so if you can be patient, there's a good chance that good answers will be forthcoming. At the very least you have posed a challenging problem for all of us to consider.  Thanks for asking.

Sincerely,

Tom

Thomas D. Sandry, PhD
Industrial Statistical Consultant, Retired

tdsandry@optonline.net

------------------------------
Thomas Sandry
------------------------------

Original Message:
Sent: 12-23-2021 07:16
From: Michael Miller
Subject: Strategies for Managing Multiple Comparison Risk in Correlated Contingency Tables

Hi Joseph,

You might investigate the work of Peter Westfall, who helped create SAS, PROC Multtest.

MIke Miller
Langhorne, PA

------------------------------
Michael Miller
Mfmillstat, Ltd.

Original Message:
Sent: 12-21-2021 10:47
From: Joseph Conklin
Subject: Strategies for Managing Multiple Comparison Risk in Correlated Contingency Tables

I have a collection of correlated r x c contingency tables.  The columns are time periods 1 to t.  In each contingency table, one column c* is a benchmark for statistical significance testing.   For any cell in a column c' not c*, the value in the cell is tested against the corresponding cell in c* for statistical significance.

In each time period, I collect a different and unique sample of size n according to a complex design.  The rows of the kth contingency table are values of a categorical variable that measures an important characteristic of the members of the sample in a given time period.

Since multiple characteristics of the sample members are represented across the contingency tables, the significance tests are not independent.  Should I control for multiple comparison risk by

(1) the Bonferroni correction,
(2) the Benjamini-Hochberg or Benjamini-Yekutieli correction,
(3) fit an unsaturated log linear model and declare significant cells in columns other than c* with extreme standardized residuals, or
(4) some other way?

Thank you.

J. Conklin
Laurel, MD

------------------------------
Joseph Conklin
Senior Statistician
Office of Justice Programs
------------------------------