I wouldn't worry about it.
How to put this....
okay here goes..
A lot of ppl have fantasies. It's something I feel you generally would keep to yourself though. Like lets say you had a blonde hair gf and say you have fantasies about a brunette, you're sure as hell not going to tell your gf about this are you. Do you get my point.
I have a bf and i too have an imagination but he will never know most the things i think about. it's personal..
There's nothing wrong with it though..
You prob think of your friends mother because you see her a lot & find her attractive. no harm done. don't be embarressed but maybe keep try & keep it to yourself.