ASA Connect

Dear All,

I’m having an internal argument between myself and me over whether an analysis is or is not valid and I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer.  

I’m very concerned about the validity of the analysis described below because of low sample size and because the dependent variable is the sum of the 5 different independent variables. 

The study participants are grouped into three groups.  The sample size for the pooled participants is greater than 30 but not by much.  Consequently, the sample sizes in the groups are low.  Other than coding of Likert responses, I haven’t done any weighting or other manipulations and transformations of the data.

The participants complete a 24 item Likert scale questionnaire.  Each Likert item has 5 possible answers, coded 1 through 5.  The 24 items are grouped into 5 subscales and, for each participant, a subscale score is calculated by summing the coded answers for the questions in each subscale (5 subscale scores per participant).  Subscale 1 has 7 items (range of possible subscale scores, 7 to 35), subscale 2 has 6 items (range of possible subscale scores, 6 to 30), subscale 3 has 5 items (range of possible subscale scores, 5 to 25), and subscale 4 and 5 each have 3 items (range of possible subscale scores, 3 to 15).  Finally, an overall score is calculated for each participant by summing the answers on all 24 questions (or by summing the 5 subscale scores).

I have a hypothesis (more of a hunch, really) that, for different groups, scores on different subscales are differentially “contributing” to increasing the overall score for the questionnaire as a whole.  In other words, in group 1, subscale score 4 may be the main “contributor” to the overall Likert questionnaire sum.  However, in group 2, subscale scores 1 and 4 may be the main “contributors” to the overall Likert questionnaire sum.  In group 3, subscale scores 1, 2, and 4 may be the main “contributors” to the overall Likert questionnaire sum.  

To test my hypothesis, for each group, I ran five separate bivariate Pearson correlation analyses (a total of 15 analyses all told) for each combination of the one of the scores on the five subscales (5 independent variables) and the single overall score (dependent variable).  The correlation coefficients support my hypothesis/hunch that different scores on different subscales are differentially contributing to the overall score for the questionnaire as a whole.  Depending on group of participants, not all subscale scores are correlated statistically significantly with the overall score.  In other cases, depending on group, some subscale scores are marginally statistically significantly correlated (p values between .01 and .049).  I’m not convinced that these are real differences so I’m tending to discount them.  But, in some cases, the correlations are highly significant (rho > .9, p values < .005).  These might be real differences.  Also, the patterns of high significance differ between groups (i.e., a given subscale score may be highly correlated with the overall score for one group but not for another group).  This tends to support my hypothesis/hunch.

However, I’m very concerned (approaching convinced) that this analysis is not valid because (1) the sample sizes within the participant groups is so low and (2) the dependent variable (the overall score for each participant) is a sum of the five independent variables (the scores on the five subscales for each participant).  On the good side: The answers of participants are independent observations.  The post hoc effect sizes and the post hoc power of the statistically significant results are high.  The measurement scales of the variables are appropriate for the test.  But this nagging doubt continues…

In order to quiet the inner disagreement between myself and me, I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer, particularly for alternative and more appropriate analysis strategies if any might exist (maybe none exist....).

Thanks in advance for your time and responses.

Linda

Some thoughts about your data set and question:

1. All else equal, differences in group or individual status on a summated score are proportional to the standard deviations of the elements summed.  So one obvious point is, the scales having more items are more likely to have 'stronger influence' on the total score than are scales with fewer items. I'd suggest replacing scale scores as sums with scale scores as means of the responses.  That is, divide your scale scores by the number of items contributing to them (e.g., average_1 = scale_1 / 7).  That at least adjusts for the artificial effect of number of items.

2. Given #1, the next question is, are groups comparable in variance on the respective scales?  If so, then a given scale (summed or averaged) will contribute to summated score differences across groups more so than will a scale on which there are not variance differences. 

3. The deeper question is, do responses of the three groups conform to a common factor structure for the set of (you choose: 24 items or 5 scale scores)? 

4. Both #2 and #3 would be more confidently answered with larger samples than you have at hand.

Good luck with bringing the debate to a mutually satisfactory conclusion!

Dear All,

I’m having an internal argument between myself and me over whether an analysis is or is not valid and I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer.  

I’m very concerned about the validity of the analysis described below because of low sample size and because the dependent variable is the sum of the 5 different independent variables. 

The study participants are grouped into three groups.  The sample size for the pooled participants is greater than 30 but not by much.  Consequently, the sample sizes in the groups are low.  Other than coding of Likert responses, I haven’t done any weighting or other manipulations and transformations of the data.

The participants complete a 24 item Likert scale questionnaire.  Each Likert item has 5 possible answers, coded 1 through 5.  The 24 items are grouped into 5 subscales and, for each participant, a subscale score is calculated by summing the coded answers for the questions in each subscale (5 subscale scores per participant).  Subscale 1 has 7 items (range of possible subscale scores, 7 to 35), subscale 2 has 6 items (range of possible subscale scores, 6 to 30), subscale 3 has 5 items (range of possible subscale scores, 5 to 25), and subscale 4 and 5 each have 3 items (range of possible subscale scores, 3 to 15).  Finally, an overall score is calculated for each participant by summing the answers on all 24 questions (or by summing the 5 subscale scores).

I have a hypothesis (more of a hunch, really) that, for different groups, scores on different subscales are differentially “contributing” to increasing the overall score for the questionnaire as a whole.  In other words, in group 1, subscale score 4 may be the main “contributor” to the overall Likert questionnaire sum.  However, in group 2, subscale scores 1 and 4 may be the main “contributors” to the overall Likert questionnaire sum.  In group 3, subscale scores 1, 2, and 4 may be the main “contributors” to the overall Likert questionnaire sum.  

To test my hypothesis, for each group, I ran five separate bivariate Pearson correlation analyses (a total of 15 analyses all told) for each combination of the one of the scores on the five subscales (5 independent variables) and the single overall score (dependent variable).  The correlation coefficients support my hypothesis/hunch that different scores on different subscales are differentially contributing to the overall score for the questionnaire as a whole.  Depending on group of participants, not all subscale scores are correlated statistically significantly with the overall score.  In other cases, depending on group, some subscale scores are marginally statistically significantly correlated (p values between .01 and .049).  I’m not convinced that these are real differences so I’m tending to discount them.  But, in some cases, the correlations are highly significant (rho > .9, p values < .005).  These might be real differences.  Also, the patterns of high significance differ between groups (i.e., a given subscale score may be highly correlated with the overall score for one group but not for another group).  This tends to support my hypothesis/hunch.

However, I’m very concerned (approaching convinced) that this analysis is not valid because (1) the sample sizes within the participant groups is so low and (2) the dependent variable (the overall score for each participant) is a sum of the five independent variables (the scores on the five subscales for each participant).  On the good side: The answers of participants are independent observations.  The post hoc effect sizes and the post hoc power of the statistically significant results are high.  The measurement scales of the variables are appropriate for the test.  But this nagging doubt continues…

In order to quiet the inner disagreement between myself and me, I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer, particularly for alternative and more appropriate analysis strategies if any might exist (maybe none exist....).

Thanks in advance for your time and responses.

Linda

Dear All,

I’m having an internal argument between myself and me over whether an analysis is or is not valid and I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer.  

I’m very concerned about the validity of the analysis described below because of low sample size and because the dependent variable is the sum of the 5 different independent variables. 

The study participants are grouped into three groups.  The sample size for the pooled participants is greater than 30 but not by much.  Consequently, the sample sizes in the groups are low.  Other than coding of Likert responses, I haven’t done any weighting or other manipulations and transformations of the data.

The participants complete a 24 item Likert scale questionnaire.  Each Likert item has 5 possible answers, coded 1 through 5.  The 24 items are grouped into 5 subscales and, for each participant, a subscale score is calculated by summing the coded answers for the questions in each subscale (5 subscale scores per participant).  Subscale 1 has 7 items (range of possible subscale scores, 7 to 35), subscale 2 has 6 items (range of possible subscale scores, 6 to 30), subscale 3 has 5 items (range of possible subscale scores, 5 to 25), and subscale 4 and 5 each have 3 items (range of possible subscale scores, 3 to 15).  Finally, an overall score is calculated for each participant by summing the answers on all 24 questions (or by summing the 5 subscale scores).

I have a hypothesis (more of a hunch, really) that, for different groups, scores on different subscales are differentially “contributing” to increasing the overall score for the questionnaire as a whole.  In other words, in group 1, subscale score 4 may be the main “contributor” to the overall Likert questionnaire sum.  However, in group 2, subscale scores 1 and 4 may be the main “contributors” to the overall Likert questionnaire sum.  In group 3, subscale scores 1, 2, and 4 may be the main “contributors” to the overall Likert questionnaire sum.  

To test my hypothesis, for each group, I ran five separate bivariate Pearson correlation analyses (a total of 15 analyses all told) for each combination of the one of the scores on the five subscales (5 independent variables) and the single overall score (dependent variable).  The correlation coefficients support my hypothesis/hunch that different scores on different subscales are differentially contributing to the overall score for the questionnaire as a whole.  Depending on group of participants, not all subscale scores are correlated statistically significantly with the overall score.  In other cases, depending on group, some subscale scores are marginally statistically significantly correlated (p values between .01 and .049).  I’m not convinced that these are real differences so I’m tending to discount them.  But, in some cases, the correlations are highly significant (rho > .9, p values < .005).  These might be real differences.  Also, the patterns of high significance differ between groups (i.e., a given subscale score may be highly correlated with the overall score for one group but not for another group).  This tends to support my hypothesis/hunch.

However, I’m very concerned (approaching convinced) that this analysis is not valid because (1) the sample sizes within the participant groups is so low and (2) the dependent variable (the overall score for each participant) is a sum of the five independent variables (the scores on the five subscales for each participant).  On the good side: The answers of participants are independent observations.  The post hoc effect sizes and the post hoc power of the statistically significant results are high.  The measurement scales of the variables are appropriate for the test.  But this nagging doubt continues…

In order to quiet the inner disagreement between myself and me, I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer, particularly for alternative and more appropriate analysis strategies if any might exist (maybe none exist....).

Thanks in advance for your time and responses.

Linda

Here's perhaps another way to approach, though the (relatively) small n is a definite, substantive limitation:

Use the 5 subscale scores and do multivariate test for differences in the centroids between groups.

Evaluate the standardized canonical coefficients for the subscales to get an idea of relative importance to group differences.

Michael

Dear All,

I’m having an internal argument between myself and me over whether an analysis is or is not valid and I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer.  

I’m very concerned about the validity of the analysis described below because of low sample size and because the dependent variable is the sum of the 5 different independent variables. 

The study participants are grouped into three groups.  The sample size for the pooled participants is greater than 30 but not by much.  Consequently, the sample sizes in the groups are low.  Other than coding of Likert responses, I haven’t done any weighting or other manipulations and transformations of the data.

The participants complete a 24 item Likert scale questionnaire.  Each Likert item has 5 possible answers, coded 1 through 5.  The 24 items are grouped into 5 subscales and, for each participant, a subscale score is calculated by summing the coded answers for the questions in each subscale (5 subscale scores per participant).  Subscale 1 has 7 items (range of possible subscale scores, 7 to 35), subscale 2 has 6 items (range of possible subscale scores, 6 to 30), subscale 3 has 5 items (range of possible subscale scores, 5 to 25), and subscale 4 and 5 each have 3 items (range of possible subscale scores, 3 to 15).  Finally, an overall score is calculated for each participant by summing the answers on all 24 questions (or by summing the 5 subscale scores).

I have a hypothesis (more of a hunch, really) that, for different groups, scores on different subscales are differentially “contributing” to increasing the overall score for the questionnaire as a whole.  In other words, in group 1, subscale score 4 may be the main “contributor” to the overall Likert questionnaire sum.  However, in group 2, subscale scores 1 and 4 may be the main “contributors” to the overall Likert questionnaire sum.  In group 3, subscale scores 1, 2, and 4 may be the main “contributors” to the overall Likert questionnaire sum.  

To test my hypothesis, for each group, I ran five separate bivariate Pearson correlation analyses (a total of 15 analyses all told) for each combination of the one of the scores on the five subscales (5 independent variables) and the single overall score (dependent variable).  The correlation coefficients support my hypothesis/hunch that different scores on different subscales are differentially contributing to the overall score for the questionnaire as a whole.  Depending on group of participants, not all subscale scores are correlated statistically significantly with the overall score.  In other cases, depending on group, some subscale scores are marginally statistically significantly correlated (p values between .01 and .049).  I’m not convinced that these are real differences so I’m tending to discount them.  But, in some cases, the correlations are highly significant (rho > .9, p values < .005).  These might be real differences.  Also, the patterns of high significance differ between groups (i.e., a given subscale score may be highly correlated with the overall score for one group but not for another group).  This tends to support my hypothesis/hunch.

However, I’m very concerned (approaching convinced) that this analysis is not valid because (1) the sample sizes within the participant groups is so low and (2) the dependent variable (the overall score for each participant) is a sum of the five independent variables (the scores on the five subscales for each participant).  On the good side: The answers of participants are independent observations.  The post hoc effect sizes and the post hoc power of the statistically significant results are high.  The measurement scales of the variables are appropriate for the test.  But this nagging doubt continues…

In order to quiet the inner disagreement between myself and me, I’d appreciate any thoughts, feedback, and suggestions that anyone would be willing to offer, particularly for alternative and more appropriate analysis strategies if any might exist (maybe none exist....).

Thanks in advance for your time and responses.

Linda