I'm a married Bipolar man, so feel I can easily empathise with your friend. I would say that when he's gone quiet, he's feeling very alone, and most likely missing you very much. I have the same when my wife and children go away, even for short spells.
I know that I do function much better thanks to the love and care of my family. So if there's any similarity in your situation, then your presence is probably going to help him no-end.
I find that most of the time, my B.P. is well managed, and my specialist helps me to skip along the surface of what could otherwise be a very tumultuous emotional ride.
There are rare times though when life piles up, and B.P. takes hold. I'm coming out of a particularly bad phase at the moment, which is how I came to find your post. I've been looking for information on-line that might comfort me.
I remember when my wife and I first met, and she was in a very similar dilemma as to whether to stay with me or not. I think she was anxious about our future children from inheriting the condition from me. I decided that although I loved her very much, I needed to give her the freedom to stay or leave. I owe a lot to a friend of hers who gave the advice: "Bipolar Disorder is not life-threatening in the way that heart-disease or cancer is, and yet you probably wouldn't stop yourself from getting involved with someone with either of those conditions."
Love is always a risk. It would be no fun otherwise.