Yes, oil and petroleum products (like Vaseline) damage latex. Even if it doesn't break the condom, it can cause small holes in it, which the sperm can get through. Only use lubricants that say "safe for latex" or something like that. They usually also say they are "water-based."

Vaseline may not be the best thing to use, anyway. In the future, I would use water-based products that are specifically for vaginal dryness. If you don't see any improvement in a few months, you may want to see your doctor. There are prescription creams that work better.