See you can appear for IIT JAM for M.Sc. in Mathematics only if you have Mathematics subject in your bachelor's programme for at least 2 years or four semesters because to become eligible for admission to M.Sc. in Mathematics one need to have Mathematics as a subject in UG programme for at least two years or four semesters. There is no any restriction in subjects of 10+2 even it will not be considered to determine your eligibility for admission only the percentage of class 12th is to be considered for determining your eligibility.
You can check out detailed eligibility criteria from the link given below :-
I hope this information helps you.