There are many reasons why a woman might lose interest in sex. I would suggest talking to your ob-gyn about it just in case. Be sure to mention any medications you are taking.
If your husband accuses you of having an affair just because you are not really interested in sex, that could be a sign of some major trust issues in your relationship. It's perfectly natural for people to change and go through cycles. Some people are in the mood a lot, some people hardly ever are, and some go back and forth between both extremes. It often isn't a big deal, and can be caused by something as simple as stress.
If your ob-gyn does not find any physical cause, you could always try counseling. Couples counseling or sex therapy might help to educate your husband about all of the various things that can make people lose interest in sex as well work on the trust issue that it sounds like he might be having.