You could use a set of ruffled bloomers underneath as a bulk layer that might not be too bulky...unless of course this is for a male friend.
Another idea is using a slightly stiffer fabric...it in general will keep it's shape more easilly and will need less interfacing than it's thinner wimpier counterpart
three: you might also try, if you can get away with it, shaping it. Look up Lantern sleeve and this is what you'll find.
there are many versions of this and you could try using it as a base to get the pants to sit like you want.
4: polyfill isn't the best bet because you have to worry about it falling out and such. but if you're left to it. Try sewin batting, or Fusible, it's polyester and used in everything from Quilting to Tote bags and some applique work to add a bit of poof to things and projects.
Hope these help!