So... you just started your pill for two days and then had unprotected sex?
You need to take your pills on time for at least a week to be protected.
Continuing with your pills until you find out you're not pregnant will not harm any baby. But, you're having trouble remembering your pills to begin with, so you should probably switch to another form of birth control anyways.