I'm guessing you're in the US?
The worst they could do is turn you down. Just like trying to get into a bar underage - you can't get arrested for that, you just don't get in. If you are caught inside a bar drinking underage then there could be consequences, but even that's not a big deal - usually it's the bar that gets in trouble, not the underager.
Anyway, back to adult toy stores. In my experience the people in adult toy stores are super cool, nice and helpful. I had a guy shop worker help me pick out my vibrator years ago and he was so great at helping me choose the right one. Honestly I think even if you flat out told them you're 17 they wouldn't care - especially if you're there alone.
Here in Canada at least, the must-be-18 to go into the store rule is for a couple reasons. 1) There is pornography in plain view - even the packaging of some of the toys would be technically/legally be considered pornography. 2) They don't want a group of highschool (or younger) kids in there every day after school giggling at the 12 inch double-header.... that would be bad for business. What customer would want a group of giggling kids around when they're browsing adult toys?!?
So, being a 17 year old walking into a shop seriously looking to buy something - you will be taken seriously and they likely won't care even if they think you're under 18. You'll be taken even more seriously if you go alone - it's a private sort of purchase anyways - and then you can pay cash. =)