I think the problem with this question is that people are looking for one of two answers; either size matters, or it doesn't.
There are 3 billion women in the world and every one of them is different, is turned on by different things and has a vagina that is completely unique in width, depth, capacity and sensitivity, just like every penis is totally different. What might be too small for some is just right for others; what is too big for some is what others crave.
If you could imagine a spectrum running from "I like 'em small-ish" through "it doesn't matter one way or another" all the way to "the bigger the better" then every woman will be found somewhere along it. Well, that's my feeling (or desperate wish perhaps!) as a bloke anyway.
We try to simplify the issue and make generalisations about it and also make it into a bigger (no pun intended) deal than it really is. If a woman is turned on by a big one then fair enough but if that's all she's interested in then I'd think her pretty shallow (or completely the opposite as the case may be!) and not worth worrying about. I went out with a girl whose breasts were noticably different sizes. I didn't split up with her because of that but because she turned out to be a bit of a nutter who harboured racist views!
I would imagine size is more of a concern on a one-night stand basis as it's pure emotion-less sex where the pleasure of the other participant is, let's face it, of secondary concern. In a relationship you'll be more experimental and also do what you can to make things work if you really like/love someone.
I'd say I was on the smaller end of the average scale and one girl I went out with was definitely unimpressed by what i was packing and split up with me days after the first time we slept together. Others have orgasmed as soon as I went near them and told me I was the best they'd had. Clearly they have different opinions on the importance of size.
Look upon it as an issue of compatability rather than "does it or doesn't it matter?".