This message was deleted.
# ask-for-help
s
This message was deleted.
s
@Aaron Pham maybe?
a
U can pass blfloat16 into the model and save it with it
y
So can I understand it as, if the model in memory is in
bfloat16
, then the saved model will also be in the same format? Maybe it’s also on transformers side that, even though
config.json
specifies the model to be
bfloat16
, the loaded model is still
float32
. I will look further into this.
s
@Aaron Pham is that correct? I read your message to mean you need to pass an option into
save_model
, but I'm not exactly sure.
y
I was able to resolve it by simply passing
torch_dtype=torch.bfloat16
while loading the model in memory. The saved model is now only 13GB 😄
a
the loaded you can pass into via
kwargs
for
load_model
torch_dtype should work as well
s
Great that you got it resolved!