With that due date your doctor thinks that you ovulated/conceived around April 6. So sex from about April 2 to April 7 must have gotten you pregnant. How was your due date determined? With a scan or by using your LMP. A scan is usually very accurate around week 6. After that it gets less accurate until week 19 when it is not useful for dating any more. The LMP method is also less accurate unless you have the same length cycles every month.
With these dates, the 21 of April is definitely way to late. So either something happened to the condom on April 6, or your ex made you pregnant. It is probably safe to say that your ex is more likely to be the father. He ejaculated inside you after all. Your husband did not on April 6. Getting pregnant from sex on March 30 is certainly possible if your dating was only slightly off.
Best of luck!