Your current weight does have something to do with how fertile you are but there are many, many other factors as well. Weight is just one small factor - especially in your case because your BMI is not that high.
If you are trying to lose weight and get pregnant at the same time it is important that you work out but also eat well. Certain foods will help increase fertility and will be important to your baby once you get pregnant. Folic acid, fibre, omega 3s and vitamin D are just a few examples of things you should not be skimping on. Working out will not only make you more energetic but it may also increase your sex drive. If you can, work out with your partner so you are on the same page.
Keep trying and do not give up! Once you do get pregnant, vary your exercise routine just a bit and continue to eat well for both you and baby.
Good luck!