See, Mathematics in class XIIth examination is treated as a core subject and you cannot ever replace a core subject from your palette no matter what. So, unfortunately you CANNOT replace Maths with Physical Education immaterial of whether you want Maths or not. So, if you fail in Maths, then you'll have to give compartment examination.

Hope I have answered your question. All the best!