You should both see fertility specialists. A simple test wherein your husband's semen is tested will tell you if he's part of the problem. You will also need to be tested - they may chose to perform a hysterosalpingogram where they insert a dye into your uterus and check the flow to your ovaries through your fallopian tubes.
We waited 3 years till we finally saw the doctors and ended up doing in-vitro fertilization. We were lucky enough to have a baby boy, who is now almost 9 years old.
Good luck to you; don't wait a minute longer if you are serious about having kids.