I haven't worn lolita or ever plan on it, but I have this to say to everyone who feels uncomfortable when being stared at.
When I see someone dressed interestingly, in a way that I really like, I stare. I stare hard, whether I want to or not, completely unconscious that the person I'm staring at is uncomfortable with it. I wouldn't be surprised if others are the same as me, and are just curious about what you're wearing. Getting all dressed up isn't so fun if people don't notice your outfit, right? So why dodge glances and feel bad for it? You can shrug off a glare no problem, just keep your head up and have fun.
But that's just a lurker's opinion.