There is no reason for you to feel bad, you are not doing anything wrong. As a matter of fact,it sounds like you are going out of your way for him, and it does not make you feel good. Unless you can make peace with this and feel like an equal and desired partner in sex, it will break you up. If having anal sex is ok for you, don't give it a second thought.
Just like for women, there is a mental proccess that men must follow to orgasm. He's got to cut everything out, and concentrate on getting his level of excitement higher and higher. If he cannot do that, he will not get there. There is something about anal sex that allows him to do that. It is probably because of the taboo around anal sex
Have you tried switching roles to see if that makes a difference? Get on top of him and ride him? Facing him and facing away from him (cow girl and reverse cow girl positions) Or tie him up and ride him. Talk dirty to him? Tell him in graphic detail what you want him to do to you? See if that will make a difference