I avoid houses and restaurants where people smoke. In some cases, I really like the person, but my options to avoid illness matter more. I am a asthmatic, who tends to get a lot of sinus infections. I know my immune system is compromised due to my asthma and also due to my diabetes.
I recall reading just 30 mins in a smoke filled area is enough to see some damage to the arteries of non-smokers. Then there was a story on the ap the other day where a woman was more prone to cancer if she and her husband both smoked.
Second hand smoke does not have to be visible for you to react to it. I have a brother-in-law who smokes a lot in the house, and I cannot go over there. If I do, I pay dearly, for I cough, I choke and then I usually get nasal irritation or a sinus infection. Even if he refrains from smoking while we are there, there is smoke residue in his walls, curtains, clothing, furniture, etc......
If you are a smoker with an infant, and you never smoke in the house, but do it outside, your child still has exposure. It will be in your skin, your hair, and your clothing. I have a friend whose child got sick whenever he rode in his fathers truck, for the father car-pooled with a smoker. The smoker's clothes left deposits on the seats, despite no one ever having smoked in the cab. Ten years ago, it was a pretty severe case, but not so out of the ordinary any more.