If they are, indeed, ingrown hairs, which is what it sounds like they are, you'll likely be able to see a little bit of hair under the skin. The easiest way, as awful as it may sound, to take care of this problem is to use a pair of tweezers and try to grasp the hair with the tweezers and pull it out from under the skin. Do so lightly, however, do not pull the the hair right out of you, just our so the tip isn't covered by skin anymore. If you completely pull it out, then you're back where you started, with no hair, and chances are it'll just become ingrown again... Let me know if you have any quesitons.