We've been friends for years, and we'll go several months without hanging out, and when we do, low and behold, poof, it's all physical.
I had to finally come to terms with myself, because I was ready for a relationship, but when it came to this guy, my brain said, "uh, what? You want something with a guy that has stood you up and two days later had sex with you? Uh, no." so that feeling turned off real quick.
He wants to have a casual sex relationship with you, accompanied by the occasional friendly chat.
You have to make up your mind on this one, his mind is made up already. He wants sex and a friendship, nothing more.
Good luck.