I have to admit to being fascinated with people's faces, too. I like to study them and try to work out their biography from it, and sometimes it can get me into trouble.
I watched a group of travelling people in a bar one night and I became completely absorbed in the face of one of the men - to me it seemed almost non-human, like the face of a wood nymph or a mischievous sprite. When I went to leave, I was followed outside by two of the women in the group who pulled a knife on me for staring at their man! It took some quick thinking to get out of that one.
I find myself staring at people on public transport. I often wish I could be invisible so that I could study their faces more intently. Today I saw an Indian woman on my bus with a face that looked almost beatified. I kept glancing at her, then looking away, then back again, and then she smiled at me.
Sometimes you see a face in a crowd that seems to resonate with you. This was one of those times.
I am a people watcher, I suppose. Sometimes, though not often, I have experienced what you describe, Philo - being so irritated by a face that it almost maddens me. But a voice can be even more maddening and I find that in a restaurant I cannot sit near anyone who is talking or laughing too loudly. To me it is offensive and sometimes downright painful. I hate loud voices and people who talk too much! Perhaps I am overly sensitive, but hey, at this point in my life I can pick and choose who I want to be around - one of the benefits of middle age!