How much full fat dairy?

I have recently changed from skimmed milk products to your suggested full fat dairy. I am keen on my exercise and seek to keep my bodyfat to a minimum. I am worried that using full fat dairy will cause an increase in bodyfat and increase my cholesterol. I realise 'fat' doesn't make you fat and that managing weight is all to do with eating according to your metabolism, but there must be a limited daily intake of full fat dairy? Surely I can only afford to consume small amouts of full fat dairy? Please advise.  Neil