I don't ban junk food completely out of my diet, I just cap how much I can eat it a week. If you ban something you really love to eat, you're just going to crave it more. So just allow yourself maybe one sweet treat a week or maybe every other week can help you from binge eating or falling off the wagon. That's what I do.
However, if you really feel like you shouldn't be having any sweets at all, just don't keep it in the house, out of sight out of mind.