Unequal or imbalanced size between the pupils of the eye is called "anisocoria". It seems that you took the right right steps when you removed the contact lens because it bothered you, but it doesn’t seem likely that a contact lens can cause anosocoria. Anisocoria can be due to a neurological disorder so you should probably consult both an ophthalmologist and a neurologist. A CT-scan of the head will be probably requested.