It sounds like maybe it's not fitted exactly to your face? If there's enough room for sweat and friction to generate it may be too loose or gappy underneath, you need a decent amount of surface for spirit gum to work. It's the reason why my elf ears will stay on all day but my Amethyst gem, which is a hard, flat object, fell off constantly because the surface area wasn't constant and gapped (that area being between my boobs). I switched it to a brooch instead for later wearings.
I'm not sure what you made it out of but maybe look into re-forming it a little bit to fit more flush to your skin. Otherwise like Jei said you may just have to use elastic.
KuroNekoCon- '11 : Radcon-'12 : Akicon-'13 : Sakuracon-'13, '14, '15, '16, '17 : PAX West-'11, '12, '13, '14, '15, '16, '17 : Rainfurrest-'15 : ChibiChibiCon-'17 : RenCon '17
Shy Gal (Nintendo Fan Design)