Yes,it is possible that it is affecting you. Unless the condom broke, you are not pregnant..
You can speak to your doctor to try another type of hormones. Normally your doctor will however ask you to try a pill for longer than just a single cycle before giving up.
You can also keep a dose of emergency contraceptives handy for when the condom fails. That way you will be prepared and ready, with the smallest possible chance of pregnancy. Remember that one of you need to inspect the condom after sex to make sure it stayed intact.
Best of luck!