Hi,
Welcome to ehealthforum,
Sperms can die easily in hot environment and dry conditions within 20 minutes. Hence if you had wiped your hand dry and there was sufficient time gap between any other contact, you can be relaxed. Avoid habitual masturbations and maintain cleanliness and wash immediately with soap and water your hands and any other objects that have come in contact with semen (especially if they are used by others too).
Take care.