You may need to prime each material with different primers.
The purpose of primer is to prepare the surface for paint. When the surfaces differ but the paint needs to be the same, primer is that much more important. Thus, stick with what you know works on each surface. Various primers come in small quantities, you can get little 3 ounce bottles of gesso, you aren't obliged to buying it in gallons.