Later ultrasounds are not accurate in determining due date. Growth rates of babies varies in the womb in the later stages of pregnancy.
Go with the later due date, that way if baby comes 'earlier' you will be pleasantly surprised. If you start using the earlier due date, you'll be sorely disappointed (going overdue can be devastating!) if baby is 'late'.
Why would your doctor insist on keeping the due date? He is the doctor.