You can differentiate using the chain rule i.e., first differentiating cot^-1 and then tan^-1.

Differentiation of cot^-1(x) is -1/(1+x^2) and of tan^-1(x) is 1/(1+x^2).

So dy/dx = a.(-1/(1+(m^2).(tan^-1(bx))^2)).m.(1/(1+(b^2)(x^2))).

For a better clear picture, you can rearrange the terms as per your choice.

Hope this helps.