I agree - dump the chump!
This is SOOOOOOO much in relation to what I went through with a guy I "dated" for about three weeks. He was all interested in me, we dated a few times, got frisky, and then he pretty much disappeared when he thought it was convenient. Well strike one, two, three, and you're out!
The dealbreaker was when he blew me off to hang out one Saturday night. I was so falling in love with him as well, but it dawned on me that it was for all the wrong reasons. We still keep in contact on occaison, but nothing more (I'm not the type to keep a grudge on people - but that doesn't mean I don't forget).
Point is - you are being played. Either that or the communication part of his brain is broken. Or he thinks that you two are in an open relationship.
Did you guys talk about what you both want out of the relationship? Something tells me no.
Dump him if he's going to play like that.