Hii,

Yes, even if you have a backlog in math for your B.Tech, you can still take the GATE exam. According to the GATE regulations, you must be in your last year of study or have earned your degree.

Before enrolling in any college to pursue an M.Tech, you must finish all backlogs, including math. Thus, put in a lot of effort to clear your backlog while simultaneously getting ready for GATE. You're capable.