The skin around your nipples and ariolas is loaded with nerve endings. Even very light touches can trigger immediate response. Additionally, the skin itself has lots of tiny muscles that when tightened cause the nipples to become erect. The nerves and muscles are there to prepare the nipple for nursing, opening the tiny ducts so milk can flow when sucked upon.
The reaction you are getting is natural and normal. When gently touched (like when a baby mouth is on them) the breasts start to let down so milk can flow easily. Of course, if you aren't nursing now the erection can be a nuisance, but the alternative is nipples and breasts that are very insensitive. You are blessed with great nipples!
Consider putting a large square bandaid over your nipples if their becoming erect is too much of a problem. The bandaid will help cover them (a shame, really), and will also reduce the sensations.