Dear Anil, for M.Sc Applied Geology, you should have studied Geology for 3 years/six semesters and any 2 subjects among Physics, Chemistry, Biological Science and Mathematics. It is mandatory to have studied Mathematics at 10+2 level. Even for M.Sc (Tech) in Applied Geology, you should have studied B.Sc with Geology  as Honours/Major/Main/Equivalent subject and 2 subsidary subjects between Physics, Chemistry and Mahematics. Even for this course, it is mandatory to have Mathematics at 10+2 level. 

So unfortunately without studying Mathematics at 10+2 level, you cannot study Geology through IIT JAM.