Given the data you provided (sore and swollen toe after dark spec was extracted, little hole on the spot with no bleeding), you might be experiencing a puncture wound caused by a minor sharp object. The object is likely to have been too small to reach the blood vessels beneath the skin.
However, it might have been contaminated with either microbiological agents (bacteria or viruses) or a substance that is able to cause inflammation of the skin.

You might want to visit a primary health care physician who might be able to recognize the source of the spec. Anti-inflammatory drugs might be recommended. If the wound is contaminated then local or systemic antibiotics might be given to prevent or to solve an already existing infection.