Hello there! 

Greetings! 

There's no rule that says that you need to attend all the interviews , so you can definitely ignore the interviews of the iims ,for which you are not interested. This will not affect your future as every IIM has its own parameters, different weightages, they make separate merit list and call students separately. So you don't need to worry about it at all.

Thankyou