I'd wear my lolita clothes more often but...alas, as a guy, it's not always accepted....and it certainly isn't always accepted in the EGL community either.
If you're really serious about it, do your research and find and join or even start your own lolita group. They're a great way to be able to enjoy your fashion more often than just at conventions and from what I've heard, you can do things like swap meets, tea parties and other outings.
I'd have to say go for it, people are going to look at you funny if you do anything out of the ordinary anyway; so why not just do what you enjoy?
"People do not care. If you cannot keep up, you will lose everything."
“It’s not how tragically we suffer, but how miraculously we live.”
~Tea is wisdom, just add water~