It depends on how you see your career. Civil Services is very unpredictable. Think this way - what will be your career options if you cannot crack the exams in maximum attempts available. This is not to demotivate you. Interest in mathematics might lead you to some new heights in the field of mathematics. But pursuing BA just for the sake of cracking civil services is not a good idea. If you are persistent enough that you will crack the exam then it doesn't matter which undergraduate course you take up. There are people who have cleared the exam with a humanities background and people with a mathematics background.

All the best.