The data you provided indicate that you might be experiencing allergy. What causes the allergy is really hard to say without further tests. However, since you experience more severe symptoms after taking milk, you might avoid taking milk for at least a week. Milk is rich in proteins which might be aggravating the condition. It does not mean necessarily that you are allergic to milk. Something else might be causing the allergy in the first place. Taking loratadin (the medicine that was recommended to you is the usual treatment combined with avoiding the allergen. Smoking affects the allergy in negative way. So, you might want to quit smoking for a while, too.
Taking food that is not reach in proteins is recommended, too.