It will most definitely make a difference. You are normally only fertile for a 5 day period around your ovulation date.

Your doctor based the due date on a standard 28 day menstrual cycle with ovulation 14 days before the next cycle. That will place your fertile window at 12 Nov to 16 Nov.

The encounter with the second guy was much later than that. If your cycle was thus close to normal, the first guy is the father.