I wouldn't say that it would be labeled "bad" if the English isn't perfect, or even if the voice acting isn't perfect. I doubt anyone will go expecting to see grade-A acting by professional actors (I don't mean any offense by that).
However, if you feel uncomfortable about it, one idea is to try the Voice Acting Alliance. There are tons of people on that forum that voice act for fun as a hobby, and many of them are quite good. If you have the time, you could try holding an audition for people to try out for the parts, and then edit their recorded lines for the skit when you get them all.