If some want to stay and some don't and you have two cars, why not let one car-full leave and let the other stay? Assuming that's not an option though...
As for whether or not it's worth it, I can't tell you, since I don't know what events/panels your convention will be scheduling for Sunday and whether they are ones that you are interested in. All the cons I go to have programming running for the whole time they are open (except possibly overnight depending on the venue), so whether you will want to attend depends entirely on what you like to do at cons.
If you don't go to conventions for the programming, and you think you will have gotten your fill of cosplaying and socializing and shopping and photos done before Sunday, then no, there's no need to stay.