My wife could have written Chuckles77's post, because it's my story also. I'm 38 now, and caffeine has affected me for years, although it took me a while to figure that out. I find my sensitivity to it has increased to the point that yes, one cup of decaf can affect me for days.
My symptoms, depending on how it hits me at the moment, can range from mild irritability to dizziness, restlessness, inability to lie still at night, mild social paranoia, compulsiveness, irrational thoughts, trouble concentrating, self-consciousness, etc.
Get it out of my system, and I'm fine - relaxed, happy, stable, outgoing and at peace.
I'm really embarrassed about this, because yes, people do think you're nuts...coffee...doing all this? It doesn't help that I absolutely love coffee. For the sake of my own quality of life, as well as for my wife and kids, I'm beginning to take it more seriously and I'd encourage anyone else with this problem to do likewise.