The subject of study of physics is "physical quantity" which is defined as a number with a unit. Physical quantity doesn't have to be a "physical" quantity. So physics does not study exclusively physical objects. I think this is how mathematics and physics are related, mathematics does not deal with units (except unity).