Mailund on the Internet

Andrew Gelman:

By “doing data analysis in a patternless way,” I meant statistical methods such as least squares, maximum likelihood, etc., that estimate parameters independently without recognizing the constraints and relationships between them. If you estimate each study on its own, without reference to all the other work being done in the same field, then you’re depriving yourself of a lot of information and inviting noisy estimates and, in particular, overestimates of small effects.

Couldn’t agree more.

See also: ESP paper rekindles discussion about statistics

Not directly related, but quite relevant.

If you are interested in statistics, I really recommend Andrew Gelman’s books and blog.

For no apparent good reason, I read an old post on p-values and re-read this comment:

John Larkin Says: 

Hi.sorry. I have trouble with the “if you repeat experiment lots of times…p value…uniformly distributed between 0 and 1″.
Is that true? If you do it lots of times do you get as many grouped around 0.0-0.01 as around 0.49-0.50?
It may be because I’m thinking of “experiments” (e.g. height of groups)…vs some statistical scenario whish uses the word stochastic – which clearly puts me in trouble.
I don’t think of pvalue as direct measure of likelihood of nul hypothesis. But if you compared two big samples (huge!) from two big groups twice (say, of height) and each experiment gave you a p-value of 0.99….I just get the feeling that these two groups might be very similar/same population…..
Cheers
JL

My answer was this

Thomas Mailund Says: 

John: Yes, p-values are uniformly distributed (under the null distribution) so you do expect to observe as many in the interval 0.0-0.01 as in 0.49-0.5.

You cannot consider a p-value of 0.99 as any kind of measure of similarity. It just doesn’t work that way.

The reason we are interested in low p-values is because if we sample from a mixture of the null distribution and the alternative distribution, then we expect more of the alternatives in the low end of p-values than we expect from the null.

Hope that helps.

Now that I think about it, this isn’t strictly true.

I still hold that p-values are uniformly distributed under the null model. So under the null model, you cannot conclude that a high p-value indicates strong support for the null model whereas low p-values support the alternative model. It doesn’t work like that.

But of course, the null model can be wrong in more than one way, and not all will show up as low p-values.

If your null model tells you that there should be a certain variance, and you see less, then you will probably see an excess of high p-values. The observations are more similar than they should be (under the null model).

You won’t see the problem as too many low p-values, but as too many high values.

If the p-values are not uniformly distributed, your null model is wrong. It can be wrong in so many ways that it really doesn’t matter why it is wrong. It is just wrong.

Hope that makes sense.

On Monday I start teaching my Machine Learning course again. I’m looking at the material for the first week right now, and I want to change it from last year.

Typically, my students will have had classes on mathematical modeling, a bit of probability theory and a bit of statistics, but experience tells me that they only have a very superficial knowledge about it. They don’t need much more for this class, but I still want to get some key points out regarding the statistics that we will be using in the class, and the last few years I don’t think I managed that well.

I don’t want to focus on modeling so much, and I certainly don’t want to discuss experiment design since the data we look at generally is just collected data that we need to make some kind of sense of, not collected to decide one theory against another.

It really is about a few points: Given the data and some generic model, say a neural network, why do we estimate the parameters in the way we do? What can we say about the accuracy of predictions? That kind of stuff.

I usually go a little bit into Bayesian statistics for model selection, but most of what they see in the class are different generic models that they estimate parameters for through maximum likelihood.

The thing is, while they generally remember how they estimate the parameters in different models when we get to the exam, they focus on the details of a particular model and rarely remember that they are essentially doing the same thing for all the models: maximizing a likelihood in a probabilistic model.

The first couple of years I taught this class, I definitely focused too much on the mathematical details in this. Going through derivations of the math, explaining how you got various posteriors from conjugate priors and such. Major fail.

I tried changing that last year, focusing more on examples, but it didn’t help much once we got to the exam.

Do any of you have experience with teaching statistics core concepts, preferably with some good examples? Care to share?

If you don’t teach this stuff, but have had classes like it, what worked for you as a student and what definitely didn’t work?

Is R an ‘epic fail’?

Something as popular and widespread as R can hardly be called a ‘failure’ in any meaningful sense, so of course the question is really in which aspects R is inferior to alternatives.

For most users who need a bit of data analysis, it is probably a poor first choice. R is a programming language with a lot of statistical and data visualisation support, but it is a programming language.  If you don’t want to do any programming, don’t muck about with R!  There are lots of visualisation tools and statistical tools that are much easier to use.

Of course, without a bit of programming, you are limited to what those tools can do, so if you need analysis that is not provided, you need to either find a programmer or learn how to program, and for the latter, R isn’t a bad choice.

You can get pretty far with very little effort in R, once you have learned how to program. Now learning how to program does require quite a bit of effort, but if you need to there really isn’t any way around it.  Just like there isn’t any Royal Road to mathematics (as Euclid is supposed to have said).

Sure, as a programming language R has its idiosyncrasies, but which programming languages do not?

This one made me laugh:

They may be able to get the correct answer on a multiple choice test that asks about a critical p-value. I have lived over half a century now and discovered that life holds very few multiple choice tests.