It usually takes longer than 2 hours to get sick from a cold virus ~ so even if he passed it to you during sex, it wouldn't have affected you right away.
It sounds like maybe you touched something the brother touched or something maybe at some point earlier in the day or the day before.
Colds come from so many places. Germs on door handles. Germs in the air. On the bus. In a classroom. At work. There is honestly no telling where you could have gotten them from.
Keep your hands washed

Hope you feel better soon.