It seems as if :
a) he is using his behaviour to control you
b) he is hurting but cannot trust anyone to share his hurt and anger (except maybe some friends, who might be misleading him and supporting his destructive behaviour)
As painful as it is, you need to analyse what your relationship with him was before he began misbehaving and try to pinpoint all the possible factors that may have caused him to escalate his misbehaviour.
e.g. if he has lack of respect for you ->is it because your words don't match your actions? Is it because you have one code of conduct for yourself and another for him? You you do whatever you want and says whatever you want? If so, he might also feel that he is entitled to so do, now that he's a young adult.
Try to explore all the possible reasons. Be as honest with yourself as you can be. It's painful, but don't crucify yourself. There's no such thing as a prefect parent. Every parent makes mistakes. The important thing is that our kids understand that we're trying and that we really want the best for them and that we're in the same team as them.
Let him know you want to participate in helping him to lead a fulfillling and happy life. Let him know that you still love him, no matter how bad things have gotten, and that you apologize for the times you lost your temper.
Model the behaviour you want to see in your child.
If you don't want him to shout at you, don't shout at him even if he is shouting at you. Instead, model the behaviour you wan to see from him. Be understanding, listen, then, state your views calmly.
Praise him when he behaves well in a family setting, especially if he's behaving well in front of your other kids. You can say something like : "Thank you for being so helpful in laying out the table for dinner." to him and to his siblings, you can say, "See what a blessing big brother is?"
Never try to manipulate him into doing something just because you feel you know what's best. Instead, discuss it with him and let him know the reasons why you feel this is good. If he disagrees, listen respectfully to what he has to say. If he gets upset, back down and say," I can see this is upsetting you. Let's give you some time. Let me know when you'd like us to talk about it again" or something like that.
Don't give in all the time, you have to see which battles to surrender and which need to be fought and won (with calmness, patience, understanding, love and prayer)
Good luck. Every child is unique and it takes time to figure out what they really need from us. Don't give up, pray for him a lot, and pray that you are given the wisdom to help him in the best way possible.
For your other little ones, lay down fair rules, and mean what you say. Explain your rationale to them for punishments. THe punishment must fit the crime and must always be meted out with calm and love. Always have debrief where you reaffirm your love for them, and remind them that the punishment hurts you as much as it hurts them. Treat them all fairly.
And hug them and praise their good behaviour. One things kids crave more than anything, ultimately, is a parent's approval. I think we've all experienced that. Unless, of course, the kid has lost ALL respect for that parent. When that happens, they find approval somewhere else.
God bless,
Mother of 2 teenage sons (and high school teacher).