occasional low white blood cell count

Hello - the last 3 years, for my yearly physical, everything comes back perfect accept my white blood cell count comes back low. i have seen a hemotologist and had extensive tests done...everything has come back fine. My pattern seems to be that my count drops when I am sick or testing first thing in the morning on an empty stomach, but when I re-test when I'm healthy or later in the day, my numbers are normal. Help - this is stressing me out...could this just be normal for me? I was asked to follow-up with the Hemotologist again.....This is making me so nervous.