If it is pleather, it shouldn't fray. In which case you could just add the velcro where you want it and them trim the excess. It's hard for me to tell and there's stitching all the way around so that doesn't help me decide. Is it just one strip of fabric or is it the base gold and a lining of some kind on the back? If it's just one layer it probably is pleather. Maybe if you could test in an inconspicuous spot like the end of the part that gets covered when worn.
If it is something that frays since you don't have a sewing machine you may want to get some fray check or something. I don't know what else is similar to hat but I'm pretty sure there's other options I've seen in the past.
Hopefully that's somewhat helpful. Someone else might be able to come up with something else to fix this problem.
Tentative Debuts for 2013
Laxus Dreyar (Fairy Tail)