Physics in an integral part in Engineering. Every branch is taught a Physics course in first year and sometimes even in second year. Electrical engineering is one of the branches that heavily uses applications of Phyaics, especially concepts in Current Electricity, Electromagnetism, Semiconductor and devices etc. So, it can't be escaped.
However, while studying the course you'll be able ro get through as it's related to rest of your subjects.
Good luck.