It probably will be hard for you to let him go with them. You will probably be okay for the first day and then you will miss the baby like crazy.
But I agree that taking time off of work is not a good idea. You don't want to lose your job and get a bad reference when you look for another one.
I would try to find a sitter for those few days, but if all else fails maybe just let him go with your family. It isn't like he will be with people you don't know or trust, so it may be good for you.