When .Oni was a newborn I made her a "community" baby. So now, she'll let *anyone* hold her.
It sounds rough, but when you give him to someone else, just let him go. Let the people that are holding him soothe him.
I found that it's sooo hard to hear your baby cry, but when the person is able to soothe him and he starts laughing, it'll all be worth it!!!!
Sarah