Look, I don't think you're any more selfish than anyone else and people who point their fingers at you are jerks.
You aren't "ruining" anything. If the marriage was a happy one, he wouldn't be bothering with you. PERIOD.
Obviously the marriage is already ruined. Now... if he was a happily married man, and you seduced/manipulated him into wanting you, THEN yes i would say you are selfish etc...
Don't be hard on yourself. You are human. A human who has desires and needs, and god forbid, a LIBIDO!
I say try to relax... if this situation is causing you more pain than joy, then you need to break away from it all together.
If you can relax and actually enjoy what you are doing, I say go for it and anyone who has a problem with it... well ITS NOT THEIR LIFE.
ITS YOURS. You live it and be happy!!!