Yes, an engineering student can apply for the IIT JAM Mathematics.

IIT JAM Exam is an admission test for M.Sc. for the science stream graduated students.

To apply for JAM Mathematics only you should have completed graduation with minimum 55% aggregate if you are from General or OBC-NCL category or If you are from ST/SC/PwD category should have minimum 50% aggregate.

Along with this you should have Mathematics for at least two years/four semesters in during graduation.