Agreed, there is nothing that says you have to wear it all the time
I wear sweet lolita often and rockabilly/pinup other days, which are fairly opposite to eachother!
There aren't many physical lolita shops that i know of, the big brands have some in Paris and San Fran. Most of the time you will be ordering clothing online
Bodyline is a popular inexpensive lolita company, most of their items are okay quality and shipping is $10 for airmail no matter how much you order, which is good if you are buying heavy things like shoes.
For more local shopping, you can also try shopping secondhand. There is a fb group called Lolita Sales in English where many girls sell thier secondhand lolita clothing, often for cheaper than retail price
And you can make a wtb looking for U.K sellers. Just make sure you read the rules!
Another big place for secondhand shopping is egl_comm_sales.livejournal.com. This requires you to have an LJ account, and make sure you read all the rules carefully before you post, but you can find great deals