My best guess is that I just did something wrong lol. Autocast seems to convert most of the model to fp16 anyways and that works great, so I'll just keep using that!