I've been taking my temp for a few months now, and sometimes I get up to go to the bathroom at night too, but I always take my final wake up temp, at 6:30am.
Since you're getting up at 4 and then at 7 you should really be okay, it's recommended you take your temp after at least 3 consecutive hours of sleep, so you'll be fine.
Also, your temp is likely to be lower at 4 am, just cause it's in the middle of the night when your temp is lowest, it will naturally rise as the morning progresses, which is why even on the weekends it is best to take your temp at the same time, in your case 7am, not when you wake, even another hour sleep can raise your temp enough to give you a wonky chart.
And don't know if you're already doing it but if you chart cervical mucus too then you'll be that much more sure that you've ovulated, you won't just be basing it on your temps.
good luck!