I am in the exact same situation.
Except his ex didnt even know about me until she snuck into his house and found photos of me.
This used to cause a lot of problems as I was doing the same thing also.
After a lot of talking and stuff though, we are starting to learn how to trust each other.
Maybe you should try the same thing.