My advice is not to do it. There are lots of single men out there.
Why take a chance and have a potential mess on your hands by going after a married man?
Besides, he is just looking for some extra pleasure on the side and using you as a tool. He won't be able to give you any kind of security in life.
In the end, do what your heart says...