ASA Connect

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

I'm not a sampling expert, but if I did a sample like this I would choose the starting point with a random number, which would give the smallest categories a possibility of being selected proportional to the number in them.

Say the first 6 states had 500 relevant subjects and the sampling plan was to skip 1,000 between selections.  If you started with a uniform random number, the 6 would collectively have a 50% chance of being included.

Starting in any way but with a random number would invalidate the sampling, IMHO.

Ed

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

Every unit needs to have a non-zero probability of being sampled. For systematic sampling, this means incorporating a random start. 

It should be fine to sort by whatever information you'd like and then use systematic PSS as long as the first unit selected is randomly selected such that each unit has a non-zero chance of being selected.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

What you describe is quite common, and appropriate for producing the most accurate national estimates. Whatever you use for your sort order reduces variation in the possible samples, and thus produces a more accurate estimate (if the sort variable is correlated with the outcome measure). While you are correct that you can't get a combination of some small states both included, you also have eliminated (or at least reduced) the possibility that none of the small states is included in the sample. If in addition to national estimates, you want, say, to produce regional ones, then the sort should first group all states in each region, then within that sort you can further sort by state. This will make the number of sampled cases from a region consistent across possible samples, improving the accuracy of regional estimates.  If you want accurate state estimates you will need to increase the number of sampled units.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

It's worth noting that even when the start point and sort order are randomized, the standard variance calculations are likely to be poor estimates and can lead to incorrect standard errors and incorrect p-values.  The variance calculations need to account for the joint probabilities of being in the sample.  (Steven Thompson explains this very well in his book, Sampling, now in its second edition, and derives relevant formulas.)  You have noted that some of these joint probabilities are zero--but that's only a part of the problem.

I have frequently encountered this issue in systematic spatial samples of irregularly-shaped regions: this sounds like it's exactly the same problem (albeit in just one dimension).  The calculation of the joint probabilities can be complicated.  A quick simulation (iterate the sampling plan 50+ times and track all pairs of states) will give useful estimates.  In this one-dimensional situation it is straightforward to obtain closed-form results but they would be conditional on the sort order--over which you would have to average if you are randomizing it.  I am not aware of any source of those results; you might find the simulation approach to be quicker and easier to carry out.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

Thank you William for the book suggestion and other comments.  As much as I would like to do a simulation effort, I don't think it is all that feasible with the full scope of the project itself.  The Systematic PPS sampling that I described previously is the first stage of a three-stage process that simply selects a particular subset of financial disbursement records that will undergo a more-detailed review by means of a data call and the last two stages of the sampling process.  So the simulation effort would learn more information about the potential first stage properties, but it unfortunately will not have any of the more-detailed information that is used in the end.

And as I mentioned above in a response to @David Marker, the same sort order / Systematic PPS sampling scheme I described is applied to other stratums that are not necessarily state-level type entities.  With it being financial records, these other stratums could contain items that are simply individuals/businesses, are a county/tribal region, or some other entity type.  In total it is about a five-tier hierarchy where the ultimate goal is a confidence interval about the highest tier's financial data using knowledge gained from every subsequent tier.

But my original question was about any bias being introduced by how the data is sorted for the Systematic PPS sampling.  I don't even has access to the more-detailed data but am tasked with merely reviewing the sampling and estimation plan itself.

It's worth noting that even when the start point and sort order are randomized, the standard variance calculations are likely to be poor estimates and can lead to incorrect standard errors and incorrect p-values.  The variance calculations need to account for the joint probabilities of being in the sample.  (Steven Thompson explains this very well in his book, Sampling, now in its second edition, and derives relevant formulas.)  You have noted that some of these joint probabilities are zero--but that's only a part of the problem.

I have frequently encountered this issue in systematic spatial samples of irregularly-shaped regions: this sounds like it's exactly the same problem (albeit in just one dimension).  The calculation of the joint probabilities can be complicated.  A quick simulation (iterate the sampling plan 50+ times and track all pairs of states) will give useful estimates.  In this one-dimensional situation it is straightforward to obtain closed-form results but they would be conditional on the sort order--over which you would have to average if you are randomizing it.  I am not aware of any source of those results; you might find the simulation approach to be quicker and easier to carry out.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

It's worth noting that even when the start point and sort order are randomized, the standard variance calculations are likely to be poor estimates and can lead to incorrect standard errors and incorrect p-values.  The variance calculations need to account for the joint probabilities of being in the sample.  (Steven Thompson explains this very well in his book, Sampling, now in its second edition, and derives relevant formulas.)  You have noted that some of these joint probabilities are zero--but that's only a part of the problem.

I have frequently encountered this issue in systematic spatial samples of irregularly-shaped regions: this sounds like it's exactly the same problem (albeit in just one dimension).  The calculation of the joint probabilities can be complicated.  A quick simulation (iterate the sampling plan 50+ times and track all pairs of states) will give useful estimates.  In this one-dimensional situation it is straightforward to obtain closed-form results but they would be conditional on the sort order--over which you would have to average if you are randomizing it.  I am not aware of any source of those results; you might find the simulation approach to be quicker and easier to carry out.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.

The originally described plan seemed to cluster the sample units into k-number of clusters and then select a sample of size 1 from the k clusters, using a random starting point within the first cluster. An approach I would recommend instead is to use multiple random starts independently selected, say r-number of them with the systematic spacing being every (r*k)'th unit is selected. (These are interpenetrating subsamples. Setting r=5 could be a good, manageable choice.) Calculate your population-level estimates independently for each independent subsample, and then combine these r-number of population-level estimates by computing their arithmetic means and std. errors of the means. The within-cluster variability is extrinsic to this approach. 

(Early in my career I worked with data like this for state fish & wildlife agencies that wanted to analyze the experiences of their licensed game hunters periodically with mail surveys, using as a data frame the agency's central office collection of hunting license applications that were stored on paper pages in file cabinets. [Affordable microcomputers didn't exist yet, and some client agencies had limited or no access to mainframe computers other than what we consultants provided.] Systematic sampling could be carried out by measuring the linear length of the entire license file and dividing that length by the sample size desired, then use a ruler or caliper set to that distance to choose the members of each subsample.)

It's worth noting that even when the start point and sort order are randomized, the standard variance calculations are likely to be poor estimates and can lead to incorrect standard errors and incorrect p-values.  The variance calculations need to account for the joint probabilities of being in the sample.  (Steven Thompson explains this very well in his book, Sampling, now in its second edition, and derives relevant formulas.)  You have noted that some of these joint probabilities are zero--but that's only a part of the problem.

I have frequently encountered this issue in systematic spatial samples of irregularly-shaped regions: this sounds like it's exactly the same problem (albeit in just one dimension).  The calculation of the joint probabilities can be complicated.  A quick simulation (iterate the sampling plan 50+ times and track all pairs of states) will give useful estimates.  In this one-dimensional situation it is straightforward to obtain closed-form results but they would be conditional on the sort order--over which you would have to average if you are randomizing it.  I am not aware of any source of those results; you might find the simulation approach to be quicker and easier to carry out.

I have a work project that is reviewing the work done by another team and this question is about the sampling plan that was implemented.  For some background, the project has several population types, but the easiest to explain is for the population that state-level entities.  Using the state-level entity, there is a preference for conducting the sampling plan using the probability proportional sampling (PPS) using an ancillary variable such as a state's land area or population to get a better sense of a state's contribution to the overall population estimate.  In addition to using PPS, it has been implemented with a systematic sampling plan but the data is sorted in terms of the ancillary variable.

However, the sampling interval (i.e., skip every Kth data point) is large enough that it exceeds the combined sum of the ancillary variable for the first 6, 7, or even 8 states and that some of the states will be selected several times because the sampling interval fits their ancillary variable with at least 2, 3, or even 4 multiples alone.  As a result, the sampling plan is guaranteed to never select any combination of states from the first 6, 7, or 8 values whereas a simple random sampling plan would at least allow for this possibility.  So to tie in the question referenced in the discussion subject, is it possible that the resulting sample of the Systematic PPS Sampling plan is not considered to be representative based on the sort order used?

I would think that the Systematic PPS Sampling plan can still be used, but that the data should be sorted by other means such as in alphabetical order, ZIP code order, or even a different ancillary variable.  At least these orderings would potentially allow the states with the lower values of the primary ancillary variable to be selected at the same time.

Any comments or other ideas would be greatly appreciated.