Research

[ValleyGal]

I know some of you have your master's degree and likely required research courses. I just need a little help with a couple of questions for my course. I am terrible with stats and have not done them in 15 years. I'm evaluating a quantitative dissertation, and there is a question that asks "what evidence did the research team present that proves the results were statistically significant?" All I have to go on is t-scores and an alpha of .01. The t-scores are negatives, though. Here's an example: t (68) = -2.55 < 0.01. It goes on to say "Mothers' mean satisfaction scores were lower in the traditional evaluation intervention, X=16.64 (SD=5.77) than in the focused evaluation intervention, X=19.86, (SD=4.79). Those are the only numbers I have to play with. I think the "proof" is in the t-score, right? But how does the t-score prove statistical significance? Simply the fact that -2.55<0.01?

The other question I'm struggling with is how the research team ensures validity and/or reliability. They used random assignment, so that's part of it, but what else am I looking for? I know what validity and reliability are, but don't know how they are ensured in the design. Can you recommend what kinds of things I'm looking for? I have searched my textbooks, supplemental readings, and even dug out my old stats book, but can't find anything to help.

Would very much appreciate help, if you can! I'm desperate!

 

[tango]

I guess my stats must be rustier than yours, I'm struggling to even remember what a t-score even represents! Maybe you give things different names in your area, the best I remember was working with either 95% or 99% confidence intervals.

I'm not sure if someone with a handle on statistics as rusty as mine is going to be much help here but for what they are worth my thoughts on validity and reliability are that you'd be looking for measures to control out factors unrelated to the difference the study was trying to isolate. To give a silly counterexample if you showed that the President's approval ratings were 83% in 2017 and 28% in 2020 you'd apparently be looking at a huge drop, but if the sample in 2017 were taken from rural Kentucky and the sample in 2020 taken from central Seattle it wouldn't take a genius to figure your samples were badly selected. In this case, depending on the nature of the focused intervention, you might want to consider the age, socioeconomic status, race, education level etc of the participants to make sure you didn't have, say, lots of wealthy mothers in one group and lots of impoverished mothers in the other.

Feel free to disregard all of the above if I just demonstrated my stats are even rustier than I thought!

 

[ValleyGal]

Thanks, Tango! Yes, that actually really helps for validity and reliability. I kind of wondered why they included socioeconomic status when it has no direct affect on the outcomes. But I can see now that was used as a control measure for validity. Awesome!

The alpha is set to .01, so 99% confidence. This score is set for statistical significance, but because it's more rigorous than .05, does that also suggest that it is measuring what it says it will measure - iow, reliability?

 

[tango]

Thanks, Tango! Yes, that actually really helps for validity and reliability. I kind of wondered why they included socioeconomic status when it has no direct affect on the outcomes. But I can see now that was used as a control measure for validity. Awesome!

Sometimes it's hard to see how some factors make a lot of difference but if you can control them out you can avoid the allegation that the research is invalid because it didn't consider it. I remember seeing some survey results about levels of satisfaction with cars in which Mercedes scored much lower than a really cheap brand and questions were asked whether the issue was that they were inherently less reliable or simply that people who paid $100,000 for a car had much higher expectations than people who paid $10,000.

The alpha is set to .01, so 99% confidence. This score is set for statistical significance, but because it's more rigorous than .05, does that also suggest that it is measuring what it says it will measure - iow, reliability?

Gotcha, I don't recall that being referred to as alpha back when I was doing statistics - we just called it 99% confidence. What you're looking at with confidence intervals (and I may be oversimplifying here) is the probability that what you're seeing isn't just down to chance. The higher the confidence, the lower the probability that it was just a fluke.

If you're measuring something that's largely subjective, like how satisfied people are with a process, you need to consider how easy it is to please individual people - there's always the kind of person who would complain that the gold bar you gave them was the wrong shape, and there's always the person who expresses high satisfaction even if they've been treated very badly. Controlling for that kind of variation is tricky (especially when people can shift depending on their mood that day), so you'd probably want to go for a higher level of confidence to cater for that.

99% confidence essentially means there's a 1% probability that the results you are seeing are caused by chance and 99% probability that the results are caused by whatever you have changed between the experimental group and the control group.

 

[ValleyGal]

99% confidence essentially means there's a 1% probability that the results you are seeing are caused by chance and 99% probability that the results are caused by whatever you have changed between the experimental group and the control group.

Right, but how is that proven?

 

[tango]

Right, but how is that proven?

If you'd asked me that 30 years ago I could probably have told you off the top of my head. If you're looking for a mathematical proof I'd probably need to dig just as hard as you would to find it

In generic terms I guess if you're looking at a normal distribution and you know the mean and standard deviation you can figure out how far you've moved from the mean, how many standard deviation units that represents, and work out a percentage that way. If you have a lop-sided normal distribution you may have to consider skew and kurtosis, which essentially measure the extent to which the distribution is lopsided or shifted towards the tails.

I'm kinda scratching around here looking up some statistical stuff to try and scrape off some of the rustiness. It looks like in a normal distribution 68% of the data points are within 1 SD unit, 95% are within 2 SD units and 99.7% are within 3 SD units. If your experimental group shows a mean value that's shunted well away from the control group I'd reckon you can figure where they lie on a distribution curve and figure how many SD units they moved and thereby what probability there is that the move was caused by chance.

If your control group has a mean of 100 and SD of 10 you'd expect 99.7% of the samples to be within 3 SD units, i.e. between 70 and 130. Therefore you'd expect 0.3% to be outside that range, i.e. less than 70 or more than 130. That would mean 0.15% would be less than 70 and another 0.15% would be over 130. If your experimental group shows a mean of 130.00001 and an SD of 10 you can see that it shifted three SD units and hence there's a 0.15% chance that the result was due to chance. If the SD also shifted that probably complicates things but someone as rusty on stats as me probably isn't your best choice of tutor to try to explain that

 

[ValleyGal]

So the mean and the standard deviation are the numbers that prove something is statistically significant?

 

[tango]

So the mean and the standard deviation are the numbers that prove something is statistically significant?

The arithmetic mean and standard deviation tell you what probability there is that something will happen. Hence with a mean of 100 and SD of 10, 0.3% of samples would fall outside the range 70-130. Hence there would be a probability of slightly under 0.15% that you'd see a sample of 135. There will be samples that fall in the tails - if you've got a sample size of 10,000 you'd expect 15 individuals to score 130+

To get into the math behind how you know whether your experimental group is sufficiently different from what you'd expect from a normal distribution is the sort of thing I'd really need to go and dig and figure out. If your control group had a mean of 100 and SD of 10 and the hypothesis was that whatever changed with the experimental group would result in an increased score (based on whatever the score was actually measuring), you'd have a few possibilities that might indicate something significant.

Theoretically speaking, assuming whatever changed for the experimental group actually had any effect, you might find the mean shifted upwards and SD stayed the same, indicating the effect was more or less universal. You might find the mean shifted upwards and SD shrank, which would suggest an improvement in score but primarily among those who scored the lowest. If the SD dropped it would suggest it was more of an equalizer than an increaser.

I'm probably not the best person to ask about the math behind the calculations to determine confidence intervals. That said this is the kind of thing that sometimes piques my interest to the point I start reading and figuring out how it all works, so you never know

 

[ValleyGal]

Tango, thanks for all your help! I have decided it's enough to simply say the proof is in the mean and SD. I'll take a bit of a hit on the grade, and that's okay. I think my other answers and other assignments will be enough to keep my grade up enough to pass. lol. But with your help, I at least had something relatively reasonable (for a social worker - we all hate stats!) to put into the answer box. I really appreciate your help on this!

 

[tango]

Tango, thanks for all your help! I have decided it's enough to simply say the proof is in the mean and SD. I'll take a bit of a hit on the grade, and that's okay. I think my other answers and other assignments will be enough to keep my grade up enough to pass. lol. But with your help, I at least had something relatively reasonable (for a social worker - we all hate stats!) to put into the answer box. I really appreciate your help on this!

Glad I was able to help.... I was always more into pure mathematics than statistics and it was long after I finished my degree that I started to put the two together and figure there's lots of geeky number-related fun sitting behind an otherwise opaque process that spits out a confidence interval.

I have a friend who is a doctor of mathematics. Maybe I'll drop him a note and see if he can point me towards some reading. Sometimes it's good to geek out on something just for the sake of it