If it makes you happy, it can't be that bad. Make sure he trims his fingernails so he doesn't cut you, and lube is helpful. We've actually found that raw fingers are more likely to cause irritation than fingers covered in latex or vinyl gloves. Seems to be something with the natural salts and oils on skin.
And when I say "we" I mean both of us like having fingers up our respective butts.
No, I don't think you're any kind of freak at all. She likes it with one up her bung, one in her vagina, and a thumb pushing her labia against her clitoris, and gently rolling it kind of from the side like. (Direct clitoral stimulation is often too much for her.)
Season to suit your own taste. You know what feels good, because you're in there. Just tell him, and he's sure to accommodate. Especially if you reciprocate.