Well personally I can't wear lolita all the time, and if people want to complain about people because they don't wear it as their main fashion, it's kinda their problem not mine. There are several reason I can't, for one I can't afford to wear my lolita clothes all the time because I can't afford a big wardrobe full of them and so I only have a small amount of them and unless I want to wear the same thing every day, that won't work. There are other reasons as well, I can't wear it to work, or school (there are dress codes in the Biology labs) or volunteering(again, dress codes--plus the shelter dogs would wreck it!) and I have problems with my feet so I can't wear lolita shoes if I will be walking a lot or I will be hurting...
Also if people are only comfortable wearing lolita at cons, who cares? Why does that bother anyone?