Discussion: View Thread

I am working on some repeatability reproducibility project.  My understanding is that this is also called a "Gauge" or "gage" R&R study. I'm looking for papers on sample size calculation. It appears that most sample size assessments are based on simulation or basically rule(s) of thumb. I'd appreciate any insights section members may have.

------------------------------
Chris Barker, Ph.D.
2022 Statistical Consulting Chair-elect
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

See Burdick, Borror, and Montgomery's book Design and Analysis of Gage R&R Studies. Use confidence intervals on the repeatability and reproducibility to determine the necessary sample sizes.

If you're using the operator by part crossed design and can expect that the operator by part interaction term will drop out of the random effects ANOVA model, then the analysis simplifies to estimating three variances: operator, part, and error. Parts are only necessary as artifacts to be measured so the number of parts isn't very important unless a part-dependent acceptance criterion like the number-of-distinct-categories (NDC) statistic is required. That leaves the need to estimate the operator (reproducibility) and error (repeatability) variances.

If there are at least two trials/replicates, then there should be sufficient degrees of freedom to calculate a reasonably tight confidence interval for the repeatability. Three or more trials will tighten the repeatability confidence interval but that's not usually necessary. The weakness in most GR&R studies is the reproducibility estimate which is often, in historically recommended designs such as from the QS9000 Supplier Seven Pack Measurement Systems Analysis document, determined from just 2-3 operators. The resulting reproducibility confidence interval will be very wide - essentially meaningless. BB&M recommend 6-8 operators for this reason. That range delivers a reproducibility confidence interval that is about 0.5 times the point estimate to 2 times the point estimate. Still very wide but it can be sufficient. Everyone will hate you for recommending so many operators.

Good gage R&R software like MINITAB produces a custom report that can (optional) include confidence intervals on all of the relevant performance metrics.

Be aware there are many different performance metrics used in gage R&R studies. Make sure that you get prior agreement from the customer/process owner on which ones will be used.

------------------------------
Paul Mathews
President
Mathews Malnar & Bailey, Inc.
------------------------------

Original Message:
Sent: 05-11-2021 16:47
From: Chris Barker
Subject: Repeatability Reproducibility

I am working on some repeatability reproducibility project.  My understanding is that this is also called a "Gauge" or "gage" R&R study. I'm looking for papers on sample size calculation. It appears that most sample size assessments are based on simulation or basically rule(s) of thumb. I'd appreciate any insights section members may have.

------------------------------
Chris Barker, Ph.D.
2022 Statistical Consulting Chair-elect
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

Just wondering if you have ever done one of these before? 

Depending upon your client, this question might be mout. 

For example, a Gauge R&R study I did for my MS Thesis, we should have had 8 samples from each sensor under each set of conditions. At best, we had 6. At worst, we had 3. So, I ended up doing everything with 3 samples under each set of conditions. 

If you are working with biologists on DNA sequencing, or something like that, tests might be expensive or the "space" you have for your samples might be limited. So, you might say, "I need 10 samples per condition." When the scientists stops laughing, and realizes you were serious, they might tell you that the max is 3. 

With a couple, almost clients, I said I wanted 3 samples per batch of material. Because of how quickly the process went and demand for the final product, the best they could do was 2. That was then cut down to 1. This was due to the size of the sample needed. 3 samples of gas from a cylinder was enough that they couldn't sell the cylinder afterwards. 

My suggestion would be to ask your client how many samples they can spare. Then tell them 3-4 would be good. 5-6 would be better, and see what happens.

------------------------------
Andrew Ekstrom

Statistician, Chemist, HPC Abuser;-)
------------------------------

Original Message:
Sent: 05-11-2021 16:47
From: Chris Barker
Subject: Repeatability Reproducibility

I am working on some repeatability reproducibility project.  My understanding is that this is also called a "Gauge" or "gage" R&R study. I'm looking for papers on sample size calculation. It appears that most sample size assessments are based on simulation or basically rule(s) of thumb. I'd appreciate any insights section members may have.

------------------------------
Chris Barker, Ph.D.
2022 Statistical Consulting Chair-elect
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

Your question is very timely. I work for SAS Education and just finished teaching our JMP course in gauge studies. The last question one of my students had before signing off was "how do I determine how many repeats to run?"

I treat gauge study models as mixed models with crossed and/or nested factors. In my past life as a statistician in the semiconductor industry and a statistical consultant, I never ran a gauge study with just operators and parts as factors. My big insight for you is to include time effects in the model and design. 

For sample size calculations, I agree that simulation seems to be the best solution. The developer of JMP's Measurement System Analysis platform has done a lot of work in this area. She has presented at JSM and JMP's Discovery Summit. Her Discovery presentations can be found on the JMP community at jmp.com/community. Here is one in particular that might be of interest.
https://community.jmp.com/t5/Discovery-Summit-Europe-2017/Determining-the-Right-Sample-Size-for-an-MSA-Study/ta-p/36125

This link will take you to all Discovery presentations, where you can search for MSA or Laura Lancaster's name.
https://community.jmp.com/t5/JMP-Discovery-Summit-Series/ct-p/discovery

--Di

------------------------------
Diane Michelson, Ph.D., PStat
------------------------------

Original Message:
Sent: 05-11-2021 16:47
From: Chris Barker
Subject: Repeatability Reproducibility

I am working on some repeatability reproducibility project.  My understanding is that this is also called a "Gauge" or "gage" R&R study. I'm looking for papers on sample size calculation. It appears that most sample size assessments are based on simulation or basically rule(s) of thumb. I'd appreciate any insights section members may have.

------------------------------
Chris Barker, Ph.D.
2022 Statistical Consulting Chair-elect
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

Thank you. Fortunately I have JMP.

------------------------------
Chris Barker, Ph.D.
2022 Statistical Consulting Chair-elect
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

Original Message:
Sent: 05-11-2021 17:33
From: Diane Michelson
Subject: Repeatability Reproducibility

Your question is very timely. I work for SAS Education and just finished teaching our JMP course in gauge studies. The last question one of my students had before signing off was "how do I determine how many repeats to run?"

I treat gauge study models as mixed models with crossed and/or nested factors. In my past life as a statistician in the semiconductor industry and a statistical consultant, I never ran a gauge study with just operators and parts as factors. My big insight for you is to include time effects in the model and design.

For sample size calculations, I agree that simulation seems to be the best solution. The developer of JMP's Measurement System Analysis platform has done a lot of work in this area. She has presented at JSM and JMP's Discovery Summit. Her Discovery presentations can be found on the JMP community at jmp.com/community. Here is one in particular that might be of interest.
https://community.jmp.com/t5/Discovery-Summit-Europe-2017/Determining-the-Right-Sample-Size-for-an-MSA-Study/ta-p/36125

This link will take you to all Discovery presentations, where you can search for MSA or Laura Lancaster's name.
https://community.jmp.com/t5/JMP-Discovery-Summit-Series/ct-p/discovery

--Di

------------------------------
Diane Michelson, Ph.D., PStat

Original Message:
Sent: 05-11-2021 16:47
From: Chris Barker
Subject: Repeatability Reproducibility

I am working on some repeatability reproducibility project.  My understanding is that this is also called a "Gauge" or "gage" R&R study. I'm looking for papers on sample size calculation. It appears that most sample size assessments are based on simulation or basically rule(s) of thumb. I'd appreciate any insights section members may have.

------------------------------
Chris Barker, Ph.D.
2022 Statistical Consulting Chair-elect
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------

Please see these papers

• "Sample Size Calculation for an Agreement Study", Pharmaceutical Statistics, 9(2), 125 – 132.
• "Quantifying an Agreement Study", The International Journal of Biostatistics, 11(1): 125–133.

------------------------------
Jason Liao
Incyte Corporation
------------------------------

Original Message:
Sent: 05-11-2021 16:47
From: Chris Barker
Subject: Repeatability Reproducibility

I am working on some repeatability reproducibility project.  My understanding is that this is also called a "Gauge" or "gage" R&R study. I'm looking for papers on sample size calculation. It appears that most sample size assessments are based on simulation or basically rule(s) of thumb. I'd appreciate any insights section members may have.

------------------------------
Chris Barker, Ph.D.
2022 Statistical Consulting Chair-elect
Consultant and
Adjunct Associate Professor of Biostatistics
www.barkerstats.com


---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
------------------------------